Nov 23, 2017
Transcript
JA: Do you can I ask you do you do you just want to like lay out for us the chronology of your obsession. I think I feel comfortable saying it was. Has it become an obsession or just a just a dalliance?
DE: I mean.
SW: I mean or maybe you just noticed a crumbling building and ran over to stick your pen...
DE: I mean I am, I'm a contrarian. And I'm interested in alternative facts about science let's say. (laughs)
JA: Hey, I'm Jad Abumrad.
RK: I’m Robert Krulwich. This is Radiolab.
JA: And a little while back, our editors Soren Wheeler and I … we talked to science journalist named Dan Engber who got us kind of tangled up about something that we thought we knew about about the world, about ourselves…
RK: Something beautiful as I recall.
JA: Yeah, that we talked about at great length on this show.
CS: Okay.
JA: Hello?
CS: Hello?
SW: And to set that conversation up we’re gonna start with this guy.
JA: Hi is this Claude Steele?
CS: Yes.
SW: Professor Claude Steele.
CS: I’m the Lucie Stern’s emeritus professor of psychology at Stanford University.
SW: We actually had him on the show number of years ago.
CS: Some time ago, I can’t remember how many years ago. But -
JA: Yeah it’s been a - it’s been a -
CS: Uh -
JS: it’s been a long time. I’m looking at the sheet here and it says 2009.
CS: Whoa!
JA: That it was -
CS: That blows my mind.
JA: Yeah that was a long -
CS: I was a lot younger man then.
JA: [laughs] I know.
SW: And the reason that we had him on the show back then was because he had done a study in the mid-90s that just completely changed the way that we thought about the power of stereotypes.
CS: Well, my mother was white, my father was African American they - and they were very active in the civil rights movement. So you can imagine race was no distant or passing thing in in our life and family.
SA: For Claude, growing up, that was just everyday dinner table talk.
CS: Yes, exactly. [laughs] So the the topic is a sort of family birthright.
SW: But then years and years later, midway through his career as a psychology professor, Claude ran into a demonstration of the power of race that really surprised him.
CS: I got a job offer - this is in the 80s at the University of Michigan. And it was part psychology and part to administer a minority student program there.
SA: This is Claude Steele in the original program that we ran back in 2009.
CS: And um in the process I - I saw data that surprised me.
JA: What he saw was a troubling trend. Two kids would enter Michigan. One was black, one was white. They’d come in at the exact same level.
CS: Same skills. Same SAT score.
JA: So theoretically they should do the same when they get to Michigan. But, without fail or almost without fail, after one semester,
CS: The black kid was winding up with lower grades.
JA: How much lower?
CS: Pretty pretty um - pretty dramatic. At least two thirds of a letter grade.
JA: Meaning if the white kid got an A, the black kid who should be getting an A, too, is instead getting a B.
CS: That’s right.
JA: Or a B+.
JA: That’s significant.
CS: That’s significant. That’s significant.
JA: And he also by the way saw this performance gap between women and men when it came to math
JA: To the same degree?
CS: The same degree in advanced math courses it was comparable. I learned this is a national phenomenon. If if I was to walk into almost any college
class in the United States uh - I’d have a very high probability of finding exactly that.
MUSIC IN
DE: So I think it’s important to put it in context of what was going on at the time… (FADES UNDER)
SW: This is Dan Engber again and he says that that gap in achievement between black and white students that Claude had noticed, that was actually a huge topic of conversation at the time.
DE: there's a lot of discussion of what to do about the achievement gap and the familiar argument, is well this has to do with systemic racism and systemic differences in opportunities that play out through an individual lifespan. Now that seems right. It's also daunting because how are we ever going to like cure all of the socioeconomic disparities in this country…?
JA: That’s a big problem yeah
SW:. But then, in 1994, a different, and in many ways, very dangerous ideas was being tossed into this debate.
Reporter: Charles Murray - coauthor of the book The Bell Curve,
SW: When The Bell Curve came out.
TV show: The Bell Curve, intelligence and class structure in American life.
SW: The Bell Curve argued that one explanation for the achievement gap among others was genetic and IQ-based. Now, of course, it’s not. But even though there’s no scientific backing at all for genetic differences like this. The Bell Curve was still significant just because of the kinds of conversations it was creating at the time and the effect that it had on researchers in this field.
CS: Well, The Bell Curve is one point in a long history of that kind of argument …
that the difference between groups really is rooted in genetic differences let's just be frank and honest and if you really can’t admit that, then you don’t have the courage needed to be a real scientist. That's that's a thumbnail way of describing this experience that I've had of being confronted with with that notion.
SW: And this was obviously disturbing to Claude, well, first on a personal level
CS: in order to be a scientist are you supposed to actually be open to the possibility that you and your family and your whole race have some genetic limitation?
SW: But also because it was so weak scientifically.
CS: It's been very difficult, impossible to produce anything like definitive data
that the differences in test scores between groups is genetic.
SW: But while the differences between these students that Claude was seeing at Michigan, it also didn’t seem to him like it could be explained simply by their backgrounds or their opportunities.
CS: Because, you take, say, a black kid and white kid at Michigan. They both have extraordinary scores like, you know, they’re in the 98th percentile on their SATS. So, um, the background between the two kids, whatever it is, has not resulted in a difference to that moment in time. So, if going forward and taking a test, black kid gets a lower score or a lower performance in a course of some sort, then something must be happening right there, right there.
SW: Something must be happening in the moment.
CS: There was something there that people didn’t understand and that we certainly didn’t understand.
MUSIC
JA: So he figured he would start with the women in math issue. He brought a bunch of women in and a bunch of men. Sophomores.
CS: Brought them into the laboratory one at a time. Gave them a half an hour section of the graduate record exam you take if you’re a math major. Very, very difficult math.
JA: Mm.
CS: And sure enough, the women who had all the same credentials coming into that situation performed dramatically worse than the men.
JA: Worse as in -
CS: It’d be a couple hundred points on an SAT test.
JA: Big difference.
CS: So this was - this was a big effect.
JA: So Claude Steele thought all right - step one complete, I’ve got a lab situation that resembles the real world, good. Now the next step is to tweak things a little bit, see if I can mess around with it.
MUSIC
JA: Now, normally in these situations -
CS: The test giver’s got a white lab coat on, and he brings in a big stack of cellophane wrapped tests. And he puts a clock on the table - and it’s all - it’s all - you know it’s like - that’s - [laughs] that’s gonna intimidate almost anybody!
JA: Maybe that’s what’s happening, he thought. What if I took away the clock, took away the coat and most importantly, right before the test, I had the test giver instead of saying the normal - I’m going to give you a test, pre-test thing. Maybe instead, say something like this.
CS: Look you may have heard that uh women don’t do as well as men on difficult standardized math tests. You may have heard that. But that is not true for this particular test.
JA: Ohh.
CS: This particular test does not show gender differences, never has, never will.
JA: He wondered if maybe saying that simple sentence before giving the test would have an effect.
MUSIC OUT
CS: And sure enough, I wouldn’t be here if their performance didn’t go up to match that of the equally skilled men.
JA: That performance gap totally vanished.
CS: Sheee look at this thing. So we raced and did it very quickly, the same kind of an experiment with African Americans.
JA: There, the pre-test disclaimer went like this.
CS: This is an instrument that we use to study problem solving and it is not diagnostic of individuals’ intellectual ability.
JA: In other words, this is not a test of your intelligence. I repeat not an IQ test.
CS: So just do the best you can.
JA: And with that simple disclaimer at the start -
CS: Same kind of an effect.
JA: The black students and the white students were now equal.
MUSIC
CS: Just recently uh Ryan Brown and Eric Day did a even clever-er treatment. They - there is an IQ test, which is nonverbal.
RB: It’s called the advanced progressive matrices.
CS: It has figures.
ED: Very abstract, they got lines crossing.
CS: That you have to match and so on.
ED: Checks uh -
RB: It’s essentially pattern matching.
ED: Diamonds with dots in em. It’s -
RB: Totally visual.
ED: Yeah it’s very -
CS: And so they could represent that test -
As it is, as an IQ test, it’s in fact seen as the gold standard of IQ test because it’s quote culture free.
ED: There’s no math, there’s no reading.
CS: Because it doesn’t involve language. Uh or you could represent the exact same test as a puzzle.
RB: Puzzles.
ED: Puzzle.
JA: Meaning you can give an IQ test to a bunch of kids and the black students will perform worse. But if you give that same test - lose the word test, lose the word IQ, and just call it a puzzle?
RB: The black participants suddenly jump up in their performance.
ED: Basically we got a reversal.
CS: When you represent it as a puzzle, blacks perform as well as whites.
RB: They - they did, yeah.
JA: That’s all it takes. Just change a few words.
JA: Stereotypes are powerful. Okay, that makes sense. That - but in terms of understanding how this works, can you make this tactile for me? Like if the stereotype that’s having all these effects is a thing - like a -
ED: Mmhm.
JA: Like a little gremlin that bites? Like when in the test taking process, does it actually like do its damage?
ED: That - that’s
JA: When does it actually -
ED: gonna - that’s gonna be way open to debate.
JA: What does seem to be clear from the data, according to Eric Day and Ryan Brown and Claude Steele, is that the gremlin only seems to appear when the test is sufficiently hard.
CS: If the test is easy, it’s important to point out uh these effects don’t happen.
JA: It’s not that the gremlin is not there.
RB: Well he walks in with you, but he doesn’t speak necessarily until things get challenging.
CS: As soon as it - the test gets difficult,
RB: That’s where the voices kick in.
JA: Which means that for most of the test, everybody’s doing about the same. It’s only at problem number 17. The one about… cosines and factorials and whatever, where things start to go wrong, and - at least that’s the theory. At that problem, the black student starts to stiffen up a little bit.
CS: That’s right.
JA: And Claude Steele’s measured this.
CS: Their blood pressure’s elevated, their short term memory is impaired.
JA: It’s that flicker of frustration through their body that wakes up
MUSIC IN
JA: the gremlin who starts - to whisper in their ear.
RB: I don’t know if you can do this.
CS: Oh shit, is what they say about us true?
RB: They don’t think you can do it.
JA: All the usual stuff. And even if the student doesn’t believe it, which is likely -
RB: See you don’t have to believe it, that’s the kind of insidious thing here.
MUSIC OUT
JA: Just the fact that he has now this extra bit of mental chatter.
RB: That little guy whispering.
JA: Well it’s a distraction.
CS: And that makes their performance go down.
JA: Just a little bit.
RB: All of this dialogue is keeping you from being 100 percent focused on the task at hand which is solving these problems.
JA: So the real subtle power of a stereotype isn’t that it prevents you from doing a thing you want to do. It distracts you for just a beat from doing the thing you want to do and that may be all the difference.
MUSIC IN
SW: So that’s how we ended the piece that we did back in 2009. But in the years after Claude did that original study, the effect, which he called “Stereotype Threat,” became one of the biggest...
Male voice: Stereotype threat-
SW: and most important ideas
Female voice: Stereotype threat.
SW: in all of social psychology.
Male voice: Now some psychologists say, stereotypes can become self-fulfilling prophecies.
SW: And Claude Steele...
Male voice: Ladies and gentlemen it’s my great pleasure to present, Claude Steele.
SW: became a sort of academic rockstar.
Male voice: Professor Claude Steele.
Female voice: Dr. Claude Steele.
SW: Speaking to overflowing audiences at places like Columbia and Cornell.
Male voice: Welcome Dr. Steele.
Female voice: Dr. Steele.
SW: And this idea of a stereotype threat was shown to be relevant in cases that had to do with age and socioeconomic status. There were studies about women playing chess, men being tested on like social sensitivity. I mean, Claude’s work ended up inspiring sort of a whole generation of social psychologists.
MI: Yeah, I would say that the original stereotype threat paper by Steele and Aronson blew me away. It just spoke to me and was beautiful and it seemed to offer answers, you know, to questions that troubled me.
SW: So this is Michael Inzlicht...
MI: a professor of psychology at the University of Toronto.
JA: So when did you begin -- if you could give us sort of the cliff notes version of your history with stereotype threat in particular?
MI: I mean I think... I was certainly attracted to that that part of social psychology that dealt with prejudice and discrimination. so I’m Jewish, I went to, you know, Jewish day school and high school and kind of perhaps baked into me was a desire to, for social justice, um, and you know seeing the evils of prejudice and how those evils taken to their logical extreme, what could happen. So I was passionate about the topic. I, um … so it seemed like a very hopeful, sort of explanation that also offered relatively easy solutions to fix.
JA: Sounds like you really came into it to with a very social, political sort of bent to it.
MI: Yes, that’s right. I wanted my work to have an impact. I wanted it to, yeah, to change the world.
JA: Coming up, after the break, the Michael tries to change the world but the world kind of changes…
RK: him…
JA: Him, yeah…
JA: Jad
RK: Robert
JA: Radiolab. We’re back and we’re talking about Claude Steele’s seminal research stereotype threat: how the threat of negative stereotypes can impact a person’s academic performance.
SW: Right and uh just before the break we were talking to Michael Inzlicht who, after getting into stereotype threat research, he went onto grad school and really focused in on stereotype threat as his field of choice.
MI: Yeah, so…
DE: How many studies of stereotype threat did you end up doing?
MI: I would say in the order of, like, in terms of numbers of studies 20 maybe? 15 to 20? - something like that?
SW: He did a lot of studies on women's performance in math but also just looking at different environments and how they create or encourage these stereotypes. And over the course of his career he ended up editing a book about stereotype threat.
MI: Oh, well yeah… yeah, yeah...
SW: Gave numerous talks on the subject.
MI: All over the world
SA: He even signed a brief, uh - kind of explaining stereotype threat
MI: to the Supreme Court of the United States -
Male reporter: The Supreme Court heard the oral argument in Fisher v. University of Texas at Austin -
SW: As it applied to the question of affirmative action
Male reporter: The court will decide whether race used in university and college admission policies is constitutional.
MI: Um so I - you know put my name to this.
SA: But then -
MUSIC IN
SW: But along the way he says, he noticed some things that troubled him.
SW: And weirdly enough it all started he says, when he moved to Toronto.
MI: I started my - my current job at the University of Toronto in 2005 -
SW: And while he was there he was running a lot of his stereotype threat research on women in math you know giving women a test and then doing some intervention to reduce the stereotype and seeing if there would be a difference in their test scores
MI: And, simply put -
MUSIC OUT
SA: For the first time -
MI: I was not able to consistently show any effect of, of, of stereotype threat.
JA: Hm.
SA: In other words, the women who had received this stereotype-threat-reducing intervention performed just about the same as the women who hadn't.
MI: So I think when I first got here, I failed to replicate some of these effects, and like let me go back to the drawing board. Let me, you know, think about you know - what is my - how is my population different?
SA: And his first thought was maybe it has something to do with Toronto.
MI: So Toronto uh - incredibly diverse place.
SW: like take the freshman pysch classes he was teaching.
MI: We’re talking about you know a third East Asian, a third South Asian, and only about 15% Caucasian or Eur- European Canadian. And then a little bit of everything else.
SA: So he started wondering,
MI: To what extent do our students even ha - are they aware of these stereotypes? About, you know, women and math? Um, and remarkably, when I would ask, you know, our students this
SA: Like he'd ask, you know, ‘Who here has heard that women tend to be worse at math?’
MI: I would say no more than like a quarter to a third had you know a strong awareness of this stereotype.
SW: So he started running the experiments again but
MI: I’d only pick those women and men who actually had awareness but even then. I still then couldn’t get the effect. I was like, okay, well - maybe I’m doing something wrong,
SW: Like maybe the interventions he was designing weren’t reducing the threat or with these students the threat just wasn’t that much of a threat in the first place. But by this time Michael had already done a ton of research on stereotype threat and
MI: you know, I’m I’m a person who gets bored eh rather quickly um, and I just you know, started, eh - started losing interest.
SA: But then um just a couple years later -
MI: Well, so um - things changed (laughs)
MUSIC IN
SA: On October 17, 2011…
MI: This one paper was published.
SA: It was a paper called False Positive Psychology.
MI: And this paper detailed...
SW: … how just by doing some very standard practices in psychology research…
MI: … using these techniques you are taught explicitly in grad school …
SW: you could sometimes end up with these sort of ludicrous conclusion.
JA: So you're saying there were this paper was showing ways in which experimenters could subtly unknowingly juice their results. Is that essentially what it was?
MI: Yeah, essentially…
SW: So the paper was pointing out that it becomes sort of standard practice that you know when you were researching for some effect like oh how happy people were or how well they did on a test that you would measure that thing in a couple of different ways.
MI: Which to some extent is considered good practice, right? You want to, no one measure captures your construct perfectly. So you should measure that thing in as many ways as possible. But now what if you find that it “works,” your hypothesis is confirmed with one of those measures but it's not confirmed with three or four of the other ones.
SW: And it was not uncommon practice at that point to just report the one place where you got in effect.
MI: You only report the one that worked.
SW: You could even argue that you were just dialing in exactly where this effect happens. But the paper, this false psychology paper pointed out that if you ignore the places where it didn't work...
MI: That's not really a full picture of what that data actually looked like.
SW: The data as a whole, if you looked at all of it, might not actually support your conclusion.
MI: I remember reading that. My jaw dropped. I sent this paper I circulated it to the other faculty at my department and all of us are many or many of us saw the importance of this paper. We, we called an emergency meeting for, you know -
JA: You called an emergency meeting?
MI: Yeah! I’ll never forget it. I mean it was -
JA: What does an emergency meeting look like?
MI: The faculty, and the graduate students just to discuss the contents of the paper and to see what it meant and what the implications were.
SA: And as they talked it over, they realized that in some ways probably some of their own work had fallen prey to the problems that this paper was pointing out.
MI: Yeah, I did see myself in some of this and I thought, you know, “Wow... like, you know, what has been implicated? What papers of mine but papers more generally have been implicated in the field you know writ large?”
SW: And, in fact, meetings like the one that Michael found himself in started happening at universities and conferences all over the world.
DE: Yeah, people looked at that paper and everyone thought, “Oh shit”.
MUSIC IN
SW: This is Dan Engber again.
DE: You know, this is, this is what, we’re… we’re all doing this stuff and now we know from this one paper it’s very, very easy to turn up spurious findings this way.
MI: Yeah it changed, it changed everything.
SW: In part, because a group of scientists started thinking, “Wow, maybe we really need to go back and reinvestigate some of the key findings in our field.”
DE: They call them high powered replications or just kind of doing the same thing but with more people.
SW: So they started doing these studies with bigger sample sizes and with strict rules about what data you were looking at.
DE: They say ahead of time exactly what they're going to do and how they're going to analyze the data. So you know there's no possibility of monkeying around to get the answer you want.
SW: And these attempts to replicate or reproduce, you know, major findings in social psychology and the sort of panic that went along with it seemed to be known as the replication crisis.
MUSIC CHANGE
ARCHIVAL (Female reporter): Replication - it’s the cornerstone of science.
ARCHIVAL (Male reporter): News articles this week are talking about the reproducibility crisis in science.
DE: It’s like, oh man - what? You know, what is going on here?
SW: Because as they re-examine some of these studies
ARCHIVAL (Female Reporter): The facial feedback effect
SA: … studies that you'll find in just about any psych 101 textbook. Some of the replications were failing.
ARCHIVAL (Male voice): The study you’ll be taking part in today involves…
DE: So specifically what got me really into covering the replication crisis was news about ego depletion.
SW: So this is a whole literature of studies that were all about how we sort of use up our willpower.
DE: The original study is you go into a lab and you're presented with a dish of delicious fresh baked chocolate chip cookies. I love the method section of that paper they describe baking the cookies in the lab so that the smell will be around the subjects when they come in and they put these cookies out. They say you can't have any cookies. And then they leave you alone with the cookies.
SW: And what they found was that, you know, if someone has to sit there resisting a cookie with it right in front of them and the smell was wafting up their nose as they sit there if they have to go through that it'll actually be much harder for them to complete certain kinds of logic puzzles. And the argument was:
DE: you use willpower on you know Task A, then you try to do task B and you just won't have a store of willpower in less later studies found you drank some lemonade in the meantime.
SW: Because sugar, the argument goes, replenishes your willpower.
DE: So there were sort of like increasingly bizarre elaborations of this original theory. It ended up working with M&Ms. It ended up working with cookies. It ended up...
SW: Over the years, Dan says this idea that you use up willpower in one place and have less and another just started entering all different corners of our lives.
DE: The insight of the original study was replicated again and again and again for decades.
SW: But then this group of scientists did this massive effort to replicate the original study. they had over 2000 subjects. They followed these rigorous rules about like what data they were going to look at.
DE: This is as rigorous a replication as you can get. And they just found like no effect.
SW: Basically no effect.
JA: But I guess I still don’t understand like how is that they’re finding nothing now but before they had a study that was then replicated a bunch of times in a bunch of different labs like what - I still don’t get what’s going on?
DE: Well the-, the- so you have that original cookie study. Yeah I think it's notable that no other studies that I know of used cookies. Now, I found studies where M&Ms were used so that just makes me wonder. I have no idea how that lab did their study but it makes me wonder what would would happen if I ran a lab and I wanted to reproduce this cookie finding an extended in a new direction and I kept trying it with cookies and it just never worked. Let me try it with Charleston Chews. Doesn't work. Hmm?
SW: Because it’s not hard to resist a Charelston Chew.
DE: Yeah, maybe Charleston Chews are not good candy. And I end up like... I do it with M&Ms and it works. Boom! There's my dissertation I publish a paper out of it. So now that's in literature. And so now ego depletion seems like an even stronger more valid thing because it's not just about cookies. Now, it's about cookies or M&Ms. My point is you don't really know how many things were tried in each individual lab.
JA: Do you think that might be what's happening in these labs is that there's a lot of trial and error and the error sort of swept aside? And the successes are offered up and then suddenly you have one more success that bolsters the idea. Is that what you think might be happening?
DE: I think that's that is the heart of it.
SW: But whatever the problems are not with those follow-up studies, the big thing was that scientists were continuing to fail at replicating these big and, and fairly well known studies like the idea that when you smile it changes your mood or....
ARCHIVAL (TED Talk): So, I want to start by offering you a free, no-tech life hack…
SW: The idea presented in one of the most watched TED talks ever, that the way you sort of hold yourself or stand…
ARCHIVAL (TED Talk): So you make yourself big, you stretch out, you take up space, you’re basically opening up - it’s about opening up.
SW: The idea that that could have a measurable impact on your behavior or even your hormone levels. That one also failed to replicate. And that kept happening with study after study after study. And, of course, you know, people would come back and say that the replication effort wasn’t done right or you, you didn’t really design it well.
DE: You’re seeing a lot of the researchers who have made their careers studying certain effects. They’re just not budging. A few of them are but most of them are not budging. So you just have a split forming. One researcher described it to me as like a civil war within social psychology
JA: So is stereotype threat now itself under threat. Is it one of these bodies of research that’s now being rethought?
DE: Well, no-one has yet done the big multi-site pre-registered replication that they did for ego depletion, the one that like really woke me up to this?
SW: But Dan says there have been sort of smaller scale, sort of here and there, attempts to replicate some of the studies. And some of those have failed.
DE:Some studies came out that found that, “You know what? I tried to redo, I tried to do the stereotype threat thing on a big group of students and I found that sometimes the opposite happened. When I tried to induce stereotype threat, the students did better.
SW: It’s the “fuck you” effect (laughing)
DE: Yeah, exactly, like I’ll show you.
CS: Well let me let me say this. Maybe this will help there because this is something, you know, I thought a lot about from the very beginning.
SW: So when we talked to Claude Steele about this, he had a couple things to say. First of all:
CS: This research has been dramatically well replicated.
SW: The stereotype threat effect has been demonstrated, you know, way more times and in way more contexts than really any of those other social pyschology studies.
CS: I don’t know if there is another, if there’s another phenomenon that has produced so many demonstrations. And, if you can’t replicate one of them or six of them, I don’t know, I wouldn’t… that doesn’t surprise me.
SW: And he says, you know, you could even see the failures as just information about where the effect really applies and where it doesn’t.
CS: So this is, this is science gradually getting sophisticated enough to help apply it in appropriate places.
SW: For example, he says he would only expect the effect to appear at times when the person is really invested in what they’re trying to do and thus, the negative stereotype really is threatening. On top of that, he says, the kinds of stereotypes that are actually threatening to a given group might change over time.
CS: I’m not sure that the stimuli and the procedures would have the same meaning with today’s college students that they had then. You know, social psychology is… the meanings come from the contemporary moment, the state of the culture at that time.
JA: Yeah, that is interesting actually. Like, if what this research is doing is studying an individual’s relationship to like threat… and like, that’s gonna be different depending on who you are because you’re gonna find different things threatening depending on where you are and even when you are…
CS: Yeah, I don’t think I know enough about “the culture of Black students” today versus “the culture of Black students” 25 years ago but it wouldn’t surprise me that there are some real differences. So I don’t put as much stock in the exact replication in Experiment A or B as I do in the conceptual replication.
DE: Is it possible… I mean, you mentioned there are just hundreds of studies that that kind of circle around the same idea in different ways That is, on its face, very compelling evidence that, you know, this is a robust phenomenon. But, at the same time, you know there’s people who gather all the data together and they say look, you know maybe there’s some kind of bias that slips in when people are doing this research, that they just keep trying different versions until they get something that looks like a result. Does that seem plausible to you?
CS: Boy, that’s a deeply uh I guess cynical? - you know I, I - account of, of a scientific literature this big. It would seem to - that, that - that doesn’t seem highly probable to me, it doesn’t seem highly likely to me. It’s clearly real and replicable under these circumstances. Just because they’re not everything doesn’t mean they’re not …incredibly important to… to the progress of this society.
SW: And, in fact, Claude points out that stereotype threat is pretty unique in the fact that many of the studies are not just in the lab. They’ve been taken out into the streets with real people.
CS:That's really where the, you know, the tire meets the road… is can you actually move the educational performance and commitment of real people in real school situations.
SW: And Claude actually sent us a list of several dozen studies showing that…
CS: … interventions designed to reduce stereotype threat can have dramatic and long-lasting effects on achievement.
SW: Now, we should say, there was at least one case… there was a study done in 2006 that a researcher tried to replicate in 2011.
DE: It’s just much smaller than the original 2006 effect. He did it again with more students this time and he got pretty much nothing. This is the same guy in the same school system trying to do the same study with, you know, hundreds and hundreds of kids and he came up with nothing
JA: I’m curious to know - I mean given that replication has become a conversation that you have to unfortunately contend with, I’m just curious if it’s changed your opinion of the work.
CS: No, I don’t think there’s anything that could make me go, “Uh oh, this whole thing is not true.”
SS: I want the truth out there more than I want anything else.
SW: This is Steve Spencer. He was an early collaborator with Claude Steele, especially on those studies involving women and math tests.
SS: I did recognize in some of the critiques real issues that we need to deal with.
SW: But Steve, like Claude, is very confident in, in the results of his studies and the effect of the stereotype threat in general. So much so that...
SS: I'm writing right now an article where I'm going to disclose every single study I've ever done, what the results are and put the data up for everybody to look at. I will admit to the questionable research practices that I've done and be as forthcoming and honest with everything in my own lab. In addition to doing that, I've entered into an adversarial collaboration it's called…
SW: Which just means he’s going to do a big scale reproduction study.
SS: … with people who have serious doubts about whether stereotype threat is real. You know, I can't say ahead of time what my reaction will be. I think what I can promise is that I will take the findings very seriously and I will do my utmost not to be defensive about them.
DE: Are you nervous about this?
SS: No.
DE: I mean the stakes are pretty high.
SS: No no. I mean, you know, what do you mean the stakes are high? I mean I'm a full professor. I have tenure. What are the real stakes for me in this? Not really much.
SW: But…
MI: Well….
SW: Well, not everyone in the field is these kinds of niggling doubts so well.
MI: There are so many pieces of evidence that things are not alright.
SW: This again is Michael Inzlicht, the professor from Toronto.
MI: To be faced with the probability, the likelihood that all this might have been for naught… or much of it might have been for not. It's you know it's unsettling. It's it's a loss of meaning.
MUSIC IN
MI: You know was I was doing good work was a controlling knowledge. I was a part of the problem was I chasing you know signal or was I chasing noise. I mean, I think the effect is, you know, it might be there. But it might be so small as to not be, not be meaningfully important.
JA: Huh, that’s interesting. It’s hard to know where to stand on this.
SW: Well, one thing, that, you know, we should make clear is that stereotypes can be really damaging. I mean, having someone tell you you suck at something when you’re under the gun to do it, that’s always… that’s gonna have an effect. I guess the question is…
JA: Yeah.
SW: … how, you know… what effect exactly and when and how and what can you do about it? And those things feel like maybe… I guess it feels like the part of me that wanted this to be a kind of very simple fix that would work everywhere and, sort of, save the world… you know, that part is, you know, is feeling a little bit of a loss--like worrying that his whole thing is shrinking on me a bit.
JA: Yeah, totally. It does feel like we’re all growing up a little bit. You kind of just have to walk away from those big simple promises.
SW: Maybe that’s exactly what you need in order to be able to find the smaller places where you can have an effect, you know? Right here, right now, with this person trying to do a particular thing.
MUSIC IN
JA: Producer, editor Soren Wheeler.
RK: This piece was produced by Simon Adler and Amanda Aronczyk.
JA: Ok, I guess we should go?
RK: Yeah.
JA: I’m Jad Abumrad.
RK: I’m Robert Krulwich.
JA: Thanks for listening.
