Adam Ruins Everything (2015) s02e15 Episode Script
Adam Ruins
1 (KNOCKING) (DOOR OPENING) Thanks for letting me use the lab on the weekend, Dr.
Sherman.
Of course, Winnie, we want all the students competing in this year's West Haven Science Competition to feel supported.
How's your research project coming along? Oh, it's very promising.
I'm gunning to win this thing.
Oh, great, well, good luck.
I'd offer to stay and help but Sunday is my fun day.
So, I'm just gonna leave you here to use the lab completely unsupervised.
Thanks.
Let's get to work, Carl.
- (CARL SQUEAKS) - (WINNIE) If my hypothesis is correct, my project will blow everyone else's out of the water.
And then I'll get into the university of my choice and be on the path to becoming a groundbreaking scientist just like my heroes.
I'll follow in the footsteps of Marie Curie and Mimi Phipps Clark.
And do research to benefit mankind.
Because Carl, in a world filled with ambiguity, science is always right.
Actually Science gets it wrong all the time! Mother physics! Wait, are you Adam from TV? Oh, you watch the show.
Yeah, when "Myth Busters" isn't on.
Now those guys are smart.
Well, wanna do the honors? Sure.
Hi, I'm Winnie Jones and this is Adam Conover, and I'm on "Adam Ruins Everything.
" Uh-oh.
Adam, I've seen your show.
It's about proving things wrong.
But you can't do that to science.
Science is how we find out the truth about the world.
Yes, it is.
But the way we do science is often flawed.
Well, not here, it isn't.
Carl and I are working on a very exciting, very solid study about how Vitamin D affects human memory retention.
Oh.
(SQUEAKING) Yes, you are a good little test subject.
Actually, if you're trying to learn about humans, he's a terrible little test subject.
(LOUD SQUEAKING) Sorry.
What? Everyone knows testing on rodents is practically the same as doing actual human trials.
Yes, everyone "knows" that.
Too bad they're wrong.
Cool! I've always wanted to be a B-A-L-B-C-J laboratory mouse.
But what are we doing here? The same thing we do every night, Winnie.
Ruining a common misconception.
(THUNDER CRASHES) Rodents make up 95% of all animals used for biomedical research in America.
That's 20 to 30 million mice a year.
Whoa, they must be such effective test subjects.
Actually, they're not.
Out of all clinical trials based on mouse models, the results don't apply to humans over 80% of the time.
Well, time to file these.
(ADAM) And those failed trials can cause hundreds of millions of dollars a piece.
And this drives up the cost of development.
Meaning we get fewer new drugs.
Yeah, we're out of funding.
(GRUMBLING THROUGH PHONE) Well, then tell the cancer patients they'll have to wait, Pete! But how can that be? I read mice share 97% of DNA with humans.
True, but that 3% makes a big difference.
Mouse metabolisms are seven times faster.
We have different inflammation reactions.
And, you know, people are over 3,000 times bigger.
- (GASPS) - (SQUEAKING) (ADAM) And our bodies can have very different reactions to medicine.
In one extreme example from 2006, a new cancer drug was tested in mice and passed with flying colors.
But when human subjects were injected with just 1/500th of a mouse dosage, within hours, all of the volunteers went into catastrophic, multi-organ failure and had to be rushed to the ICU.
Oh.
The truth is, testing on mice just doesn't tell us as much about humans as we think it does.
Jeez Louise.
So, why do we even use mice in the first place? Because they're cheap and convenient.
Compared to other animal test subjects, mice are a bargain.
And they're great in a lab setting because they're easy to tame and store and they breed like crazy.
Congratulations! It's 100! Rodents are also super easy to genetically customize.
Want a mouse with no immune system? Or how about one that glows in the dark.
Extra tumors strike your fancy? You can have whatever model of mouse you want.
Finally, the Animal Welfare Act, the only federal law that covers testing on animals doesn't protect mice at all.
(SCREAMING) Oh, yeah, we can never get away with this on a rabbit.
So, even though we know testing on mice is a problem, we keep doing it anyways 'cause it's easier? (WOMAN) And at this point, it's almost impossible not to.
Ah, Winnie, this is Dr.
Azra Raza, Director of the Myelodysplastic Syndrome Center at Columbia! I traded six Albert Einsteins for her.
(DR.
RAZA) Thanks, Winnie! And thanks for keeping me in mint condition.
Medical science is almost entirely built around mouse testing.
Researchers have been using mice since the 1800s.
And students are taught on them from day one of school.
And most other testing methods are prohibitively expensive.
You quite literally cannot do a study without mice.
At the moment in the United States, the FDA will almost never approve a trial in humans unless researchers have done it in rodents first.
The real problem is cancer research has become so mouse centric, we cannot replace rodent testing without overhauling the whole system.
Which is a shame.
Because if we want to cure cancer in humans, we should be studying human cancer cells.
We are wasting valuable resources and time on mice.
(SQUEAKING) Hey, come on! This is a family show.
(SIGHS) Thanks, Doctor Raza.
You're welcome.
And here's another tip.
If you put me in your bike spoke, it will sound like a motorcycle.
(MIMICKING MOTOR PUTTERING) (SIGHS) (BOOKS THUD TO FLOOR) Now my whole project isn't useable.
Well, you don't have to throw out all of your work.
Nope! I am gonna start a new project from scratch.
I mean, might as well get used to it.
When I'm a scientist, I'll be doing nothing but science all day.
Not quite.
Want me to tell you about it? Coou have found a cool kid to mess with? (WHISTLING THEME) You know, Winnie, the reality working as a scientist might not be quite what you expect.
Very funny, Adam.
You're a real Humphry Davy.
The chemist who discovered the effects of laughing gas? Hello? I know exactly what life as a scientist will be like.
I'll spend my days investigating the big unanswered questions.
I'll follow the research wherever it leads, beholden to nothing and no one but the pursuit of pure scientific truth.
That's a beautiful dream.
But in reality, what science you get to do depends a lot on who gives you money to do it.
'Cause if you wanna pursue scientific truth, first you gotta find someone willing to pay for it.
Around two-thirds of basic research is privately funded.
But here's the problem.
The priorities of the people handing out the funding often affect what research gets done.
(ANNOUNCER) Welcome to "Smart Tank.
" Today the Smarts will hear a proposal from Winnie, a young researcher competing in the West Haven Science Competition.
Hello, Smarts.
Uh, today, I'm asking for funding for my biomedical research.
Here's my proposal.
You'll find it's very exciting and worthwhile.
First up, a major funder of research, corporate America.
Winnie, we might be able to help you.
I just have one question.
Is this gonna help us sell more candy bars? No Why? Well, corporations and trade groups tend to fund research that will make them money.
In 2011, the National Confectioners Association paid for a study that implied eating sweets actually makes kids skinnier.
If you can't provide that kind of return on investment, I'm out.
Well, I see plenty of upside.
Your research can help us develop new drugs.
Really? Sure.
And if your results make our drugs look good, we'll publish them.
Otherwise we'll bury them so the results will never be seen.
(CHUCKLES) Yeah, they actually do that.
Well, that's wrong.
I won't be a part of it.
Oh, an idealist.
I'm out.
Well, there's two Smarts left.
Okay, another major source of funding is philanthropy.
Winnie, I apologize for my robber baron friends here.
You and I, we're in it to help people.
See? This lady gets me.
Sure, and philanthropic donors fund some worthy research.
But they're also more likely to give money to causes that have affected them personally.
You have to use this to find a cure for diamond allergies.
(LOUD SNEEZE) Or maybe I could study diabetes or sickle cell anemia? Oh, sorry, sweetie, I'm out.
This is a travesty.
Isn't anyone willing to fund objective, independent, unglamorous research? Yes, there is one final funding source.
Public funding.
Also is Mark Cuban here because I would, like, die if I met Mark Cuban.
Because government funding isn't controlled by profits or the whims of one person, it supports the unsexy yet vital basic research that all science builds upon.
And it's led to advances like Heart rate monitors, the Human Genome Project, audio recording, water purification, satellites, solar panels, super computers, GPS and the internet.
Just to name a few.
Even Google got its start with a government grant.
But you won't see that on the first page of search results.
Wow, sounds like public funding is awesome.
Finally a funding source with no downsides.
Well there is one little wrinkle.
Federal science budgets are controlled by politicians.
I hereby award you, Winnie, this government grant for (WHISPERING) What's that, Senator? You'd rather have tax cuts than scientific knowledge? And there's a powerful lobby that won't like the research? And you don't know enough about science to care about the consequences? (SIGHS) Sorry, I'm out.
No! This work is worth the money.
Ask any scientist, they'll tell you.
Sorry, politicians tell me what to do, not scientists.
Maybe not the best system.
But science needs the money.
Yes, it does.
Private funding will never be enough to support research in the public interest.
The kind of long-term careful science that doesn't generate profits or glory, but improves people's lives and increases our understanding of the universe.
We need public funding for science.
But in the past 50 years, the portion of the discretionary budget that government spends on research and development has been cut nearly in half.
And with less funding, there's more competition for fewer resources.
In the past two decades, 30% of National Health Institute grants got funded.
But today, it's about 17%.
No, no, no! And from 2010 to 2013, the U.
S.
had the biggest three-year reduction of scientific research and development since the end of the space race.
(SIGHS) But if it's so hard to get funding, then how do we do science? Well, sometimes we just don't.
In a recent study of 11,000 scientists, half of them said they've had to stop doing valuable research partway through because of economic pressures.
Who knows what great advancements we lost out on as a result.
As much as we'd love to believe that the progress of science is determined solely by the best and brightest, decisions about which science gets funded have always been controlled by non-scientists.
It's all about who holds the purse strings.
That may be true but science always triumphs in the end.
Look at all these journals.
Great, factual peer reviewed research is published every day.
(SUCKING TEETH) Ah - Ugh.
- What? Nothing, just that a surprising number of scientific studies do not hold up.
(SCREAMING) - Oh! - Excuse me.
I'm using those studies for my project.
Go ahead, but there's a good chance a lot of those studies are wrong.
Adam, please, when a study has been published, that means its data is robust and accurate.
Winnie, if you'd ever read "US Weekly", you'd know that just 'cause something is published, that doesn't mean it's true.
What's "US Weekly?" Wow, you are nerdier than I am.
(CHUCKLES) Studies can be flawed for a number of reasons.
If the sample size was too small, the luck of the draw could've skewed the results.
We should test this on 10,000 people but I only know ten.
That's okay, making friends is hard.
(ADAM) Or maybe there was a confounding variable.
Like, if the subject was using another product that could affect the results and the researchers didn't account for it.
Oh, wow, this pill is really boosting our subject's resistance to sunburn.
(ADAM) And something, very rarely, unscrupulous researchers will fake their results.
What if we just make up numbers that show that this pill cures cancer? Hello? Nobel Prize committee? I've got some data you've gotta see.
Sure, so we just need a way to find out which studies are accurate.
We already have one.
And that's to reproduce them.
To show the results weren't just some fluke, someone else has to do another study that tests the same question to see if they can get the same findings.
It's like double checking the answers.
Curse, my results were way different.
And here's the problem.
Since studies are usually reproduced after they're published, that means they have a chance to influence other research or even become famous before we find out they're wrong.
For instance, remember that famous study on power posing.
Uh, it's only one of the most watched TED Talks of all time.
It proved that by standing in the power pose, your body produces hormones that give you confidence.
Well, no one's ever been able to reproduce those results.
What? Or how about that series of studies from the '70s and '80s that showed you can actually make yourself feel happier just by smiling? Well, that makes sense, right? Sure, except that in 2016, almost 20 labs tried to replicate it and not one of them was able to reproduce those findings.
(TRAIN WHISTLE BLOWS) Holy Hawking! If that's true, then how many other studies might be wrong? Oh, a lot of them.
In fact, many scientists say that we're in the midst of a reproducibility crisis.
Like a good type of crisis, right? One research team just replicated 100 famous psychology studies but found they couldn't reproduce about 60% of them.
(BEEP) Another revisited 67 major drug studies and found that about 75% didn't match their results.
And another team zeroed on 53 recent cancer studies and couldn't reproduce 47 of them.
(BEEP, LOUD EXPLOSION) But those numbers are so high! If science gets it wrong that often, then we should be reproducing every study.
(MAN) That'd be nice, but it probably won't happen.
There and that's how you fold a fitted sheet.
This is Professor Nosek.
of the University of Virginia.
And the co-founder of the Center for Open Science.
Adam's right, Winnie.
Replication studies are not getting done.
And it's because scientists don't have enough good reasons to do them.
Wait, what do you mean? To succeed in their careers, scientists have to publish and get attention for their research.
And that's more likely to happen if their studies have new and flashy conclusions.
My study found that Fidget Spinners give you six-pack abs.
(CROWD GASPS AND APPLAUDS) And the pressure to get these kinds of results sometimes leads researchers to cut corners.
I didn't need test subjects.
I just asked my bro, Kevin.
He's super ripped.
On the other hand, replication studies make for good science.
But they don't get as much attention.
People just aren't interested in studies that don't find anything new.
And I replicated a 2003 study on classification of fine-grained polypeptide folds.
The results held up.
(CRICKETS CHIRPING, MAN COUGHS) Hey, her work is just as important.
Yeah, but it isn't as exciting.
Science is competitive.
There aren't enough jobs or grant dollars for everyone.
So, if you were trying to get people excited about your research, what would you do? Follow-up on someone's prior findings? Or try to find something new and amazing? I, uh, also did a study that found candy cures leukemia! (CROWD GASPS AND APPLAUDS) (SHUTTERS CLICKING) These incentives to publish lead to a lot of unreliable research.
One estimate suggest that in the U.
S.
, we spend $28 billion a year on biomedical research that can't be reproduced.
That's such a waste! Well, flashy findings can be useful for stimulating new areas of discovery.
But we need to change the incentives so that researchers are rewarded for producing results that are interesting and reproducible.
Thanks, Professor Nosek.
No problem.
Now if you'll excuse me, I'm gonna go use the emergency shower.
I save a ton on my water bill.
Oh, thrifty.
(CHUCKLING) So, even if my research gets positive results, I still have to find someone to reproduce it, but no one will because they only care about their own careers? Wait, what are you doing? You're right, Adam.
Science sucks, I'm done.
I'm gonna go drink some Red Bull in the woods or do whatever cool kids do.
Oh, wait! I was about to tell you why science is actually amazing! I thought the structure of my show! (BIRDS CHIRPING) You forgot this.
I don't need it.
I'm quitting the competition.
You know, I used to love that no matter how chaotic and uncertain the world felt, I could at least count on science to always be right.
But you showed me, it's not.
Winnie, do you know why science is so awesome? It's the field of human inquiry that embraces mistakes and learns from them.
Scientists don't just assert what they think is true.
They test their hypotheses against the empirical evidence.
And if it turns out they're wrong, they admit it and try again.
The fact that science seeks out its own errors and corrects them is its greatest strength.
But how do we fix what's wrong with science itself? With more science.
With more science.
When I learned about the reproducibility challenges in science, I thought we need to study this problem.
So we started The Center for Open Science to investigate why is it that researchers are not producing reproducible science even though they want to? So, we built software so that researchers can share their data more easily and be more transparent in their methods.
We also work with journals and societies and funders to align the incentives so that it's in researchers interests to be more open and reproducible in their work.
And when I first emigrated here from Pakistan, I had no idea how much research in the U.
S.
rested on flawed mouse testing.
To combat this problem, I started to collect human tissue, and this is how I found a novel treatment for pre-leukemia.
- Whoa.
- So, if something in science is broken like mouse models or faulty studies, it doesn't mean we failed.
It means we need people like you to find the solution by using Science! (ALL) Yes! You're right! I have to use science to fix science.
Eureka! Thank you, guys.
(FAINT CHATTER) And finally Winnie.
And your project is Wait, where is your project? Dr.
Sherman, I didn't do a project.
I did 19.
Because for my experiment, I replicated everyone else's research, and I found that all of their conclusions were irreproducible.
Wow, well, that doesn't quite follow the project guidelines.
But, nice work, Winnie.
Sorry, everyone else.
(MUFFLED GRUMBLING) You suck.
Well, I didn't do it for the glory.
I did it for science.
Come on, Adam, I'll show you my new research proposal.
I got some sweet new binder tabs.
Ooh, I love office supplies.
(CHUCKLING) (BOY GASPS) The card moved.
Did anyone else see that? Holy (SCHOOL BELL RINGS) Hello, we're here on set with Brian Nosek because I have way more questions for him about the reproducibility crisis in science.
Thanks for having me.
Is the crisis real and is it something that we should all be worried about? Well, it's at least a crisis of confidence.
And it's, uh, something that emerged when researchers across a variety of disciplines realized that the published literature, what we thing we're doing and producing and finding isn't as reliable as we think it is.
And so, can you walk us through a little bit of how the fields realized this was happening.
Well, the oddity is is that we've known the challenges for reproducibility for 60 years.
And it hasn't gotten a groundswell.
But that has shifted over the last ten years because perhaps the internet helped make it easier to evaluate lots of different papers at the same time.
Perhaps because of a couple of very dramatic findings that there has been a convergence.
And so it has become a thing.
Now that we know about it, we can do something about it.
That's exactly right.
Science is at its best when it is constantly self critiquing of looking at where it's limited, where it's not finding things that are reliable and looking for ways to do it better.
Every scientific claim is wrong in some fundamental way.
And the process of discovery in science is figuring out where our current models of the world are wrong and how we can make them less wrong.
I'm so fascinated by that.
Can you elaborate on every scientific theory about the world is wrong in some way.
In some way.
If we had a perfect model of the world, it would be as complex as the world.
How it is we describe how the world works, how people interact, how atoms interact, whatever it is, is a simplification.
- It is a way to try to describe - Right.
lots of different things that are happening in simple terms that make predictions about next things that will happen.
Right.
But our predictions are always imperfect.
They can't quite get exactly the phenomena as we study them.
So, they're wrong.
Right.
But wrong doesn't mean, oh, that's useless.
It's not information that we need.
It's not information that we can use.
Newton's Laws of Mechanics are the most successful theory ever, right? We use them to build bridges and skyscrapers I learned about them in high school, yeah.
And they're wrong, right? Einstein showed how they are wrong when things are moving very fast.
So, it isn't that something being wrong means, oh, can ignore science.
It means there's always something more to learn.
So, that leads me really nicely to my next question which is Is this a reason that people should stop believing or trusting science? You know, should the audience go home and every time they read about a new study say, oh, that's probably never gonna be reproduced and so I'm not even gonna think about it again? One of the big challenges of scientific knowledge is that is accumulates very slowly.
So, no one study is definitive.
Drinking coffee is bad for you.
No, drinking coffee is good for you.
No, drinking coffee's bad for you.
Well, make up your mind, scientists.
It's those claims that are initially very exciting.
But then survive confrontation over time are the ones that survive and we have confidence in.
So, as a reader of the newspaper of reporting scientific results, one thing to look for is, is this the first finding? Is this something brand-new? Uh, or is it the accumulation of a lot of different pieces of evidence? That's a very easy signal for whether that's a credible claim or a reliable claim that we know a lot about it versus something new and possibly exciting.
Yeah.
The reason to trust science is because it doesn't trust itself.
Ahh.
Right? It's that science is always questioning its own findings which makes it the most reliable way to discover things.
Thank you so much for being here.
Thanks for having me, Adam.
Sherman.
Of course, Winnie, we want all the students competing in this year's West Haven Science Competition to feel supported.
How's your research project coming along? Oh, it's very promising.
I'm gunning to win this thing.
Oh, great, well, good luck.
I'd offer to stay and help but Sunday is my fun day.
So, I'm just gonna leave you here to use the lab completely unsupervised.
Thanks.
Let's get to work, Carl.
- (CARL SQUEAKS) - (WINNIE) If my hypothesis is correct, my project will blow everyone else's out of the water.
And then I'll get into the university of my choice and be on the path to becoming a groundbreaking scientist just like my heroes.
I'll follow in the footsteps of Marie Curie and Mimi Phipps Clark.
And do research to benefit mankind.
Because Carl, in a world filled with ambiguity, science is always right.
Actually Science gets it wrong all the time! Mother physics! Wait, are you Adam from TV? Oh, you watch the show.
Yeah, when "Myth Busters" isn't on.
Now those guys are smart.
Well, wanna do the honors? Sure.
Hi, I'm Winnie Jones and this is Adam Conover, and I'm on "Adam Ruins Everything.
" Uh-oh.
Adam, I've seen your show.
It's about proving things wrong.
But you can't do that to science.
Science is how we find out the truth about the world.
Yes, it is.
But the way we do science is often flawed.
Well, not here, it isn't.
Carl and I are working on a very exciting, very solid study about how Vitamin D affects human memory retention.
Oh.
(SQUEAKING) Yes, you are a good little test subject.
Actually, if you're trying to learn about humans, he's a terrible little test subject.
(LOUD SQUEAKING) Sorry.
What? Everyone knows testing on rodents is practically the same as doing actual human trials.
Yes, everyone "knows" that.
Too bad they're wrong.
Cool! I've always wanted to be a B-A-L-B-C-J laboratory mouse.
But what are we doing here? The same thing we do every night, Winnie.
Ruining a common misconception.
(THUNDER CRASHES) Rodents make up 95% of all animals used for biomedical research in America.
That's 20 to 30 million mice a year.
Whoa, they must be such effective test subjects.
Actually, they're not.
Out of all clinical trials based on mouse models, the results don't apply to humans over 80% of the time.
Well, time to file these.
(ADAM) And those failed trials can cause hundreds of millions of dollars a piece.
And this drives up the cost of development.
Meaning we get fewer new drugs.
Yeah, we're out of funding.
(GRUMBLING THROUGH PHONE) Well, then tell the cancer patients they'll have to wait, Pete! But how can that be? I read mice share 97% of DNA with humans.
True, but that 3% makes a big difference.
Mouse metabolisms are seven times faster.
We have different inflammation reactions.
And, you know, people are over 3,000 times bigger.
- (GASPS) - (SQUEAKING) (ADAM) And our bodies can have very different reactions to medicine.
In one extreme example from 2006, a new cancer drug was tested in mice and passed with flying colors.
But when human subjects were injected with just 1/500th of a mouse dosage, within hours, all of the volunteers went into catastrophic, multi-organ failure and had to be rushed to the ICU.
Oh.
The truth is, testing on mice just doesn't tell us as much about humans as we think it does.
Jeez Louise.
So, why do we even use mice in the first place? Because they're cheap and convenient.
Compared to other animal test subjects, mice are a bargain.
And they're great in a lab setting because they're easy to tame and store and they breed like crazy.
Congratulations! It's 100! Rodents are also super easy to genetically customize.
Want a mouse with no immune system? Or how about one that glows in the dark.
Extra tumors strike your fancy? You can have whatever model of mouse you want.
Finally, the Animal Welfare Act, the only federal law that covers testing on animals doesn't protect mice at all.
(SCREAMING) Oh, yeah, we can never get away with this on a rabbit.
So, even though we know testing on mice is a problem, we keep doing it anyways 'cause it's easier? (WOMAN) And at this point, it's almost impossible not to.
Ah, Winnie, this is Dr.
Azra Raza, Director of the Myelodysplastic Syndrome Center at Columbia! I traded six Albert Einsteins for her.
(DR.
RAZA) Thanks, Winnie! And thanks for keeping me in mint condition.
Medical science is almost entirely built around mouse testing.
Researchers have been using mice since the 1800s.
And students are taught on them from day one of school.
And most other testing methods are prohibitively expensive.
You quite literally cannot do a study without mice.
At the moment in the United States, the FDA will almost never approve a trial in humans unless researchers have done it in rodents first.
The real problem is cancer research has become so mouse centric, we cannot replace rodent testing without overhauling the whole system.
Which is a shame.
Because if we want to cure cancer in humans, we should be studying human cancer cells.
We are wasting valuable resources and time on mice.
(SQUEAKING) Hey, come on! This is a family show.
(SIGHS) Thanks, Doctor Raza.
You're welcome.
And here's another tip.
If you put me in your bike spoke, it will sound like a motorcycle.
(MIMICKING MOTOR PUTTERING) (SIGHS) (BOOKS THUD TO FLOOR) Now my whole project isn't useable.
Well, you don't have to throw out all of your work.
Nope! I am gonna start a new project from scratch.
I mean, might as well get used to it.
When I'm a scientist, I'll be doing nothing but science all day.
Not quite.
Want me to tell you about it? Coou have found a cool kid to mess with? (WHISTLING THEME) You know, Winnie, the reality working as a scientist might not be quite what you expect.
Very funny, Adam.
You're a real Humphry Davy.
The chemist who discovered the effects of laughing gas? Hello? I know exactly what life as a scientist will be like.
I'll spend my days investigating the big unanswered questions.
I'll follow the research wherever it leads, beholden to nothing and no one but the pursuit of pure scientific truth.
That's a beautiful dream.
But in reality, what science you get to do depends a lot on who gives you money to do it.
'Cause if you wanna pursue scientific truth, first you gotta find someone willing to pay for it.
Around two-thirds of basic research is privately funded.
But here's the problem.
The priorities of the people handing out the funding often affect what research gets done.
(ANNOUNCER) Welcome to "Smart Tank.
" Today the Smarts will hear a proposal from Winnie, a young researcher competing in the West Haven Science Competition.
Hello, Smarts.
Uh, today, I'm asking for funding for my biomedical research.
Here's my proposal.
You'll find it's very exciting and worthwhile.
First up, a major funder of research, corporate America.
Winnie, we might be able to help you.
I just have one question.
Is this gonna help us sell more candy bars? No Why? Well, corporations and trade groups tend to fund research that will make them money.
In 2011, the National Confectioners Association paid for a study that implied eating sweets actually makes kids skinnier.
If you can't provide that kind of return on investment, I'm out.
Well, I see plenty of upside.
Your research can help us develop new drugs.
Really? Sure.
And if your results make our drugs look good, we'll publish them.
Otherwise we'll bury them so the results will never be seen.
(CHUCKLES) Yeah, they actually do that.
Well, that's wrong.
I won't be a part of it.
Oh, an idealist.
I'm out.
Well, there's two Smarts left.
Okay, another major source of funding is philanthropy.
Winnie, I apologize for my robber baron friends here.
You and I, we're in it to help people.
See? This lady gets me.
Sure, and philanthropic donors fund some worthy research.
But they're also more likely to give money to causes that have affected them personally.
You have to use this to find a cure for diamond allergies.
(LOUD SNEEZE) Or maybe I could study diabetes or sickle cell anemia? Oh, sorry, sweetie, I'm out.
This is a travesty.
Isn't anyone willing to fund objective, independent, unglamorous research? Yes, there is one final funding source.
Public funding.
Also is Mark Cuban here because I would, like, die if I met Mark Cuban.
Because government funding isn't controlled by profits or the whims of one person, it supports the unsexy yet vital basic research that all science builds upon.
And it's led to advances like Heart rate monitors, the Human Genome Project, audio recording, water purification, satellites, solar panels, super computers, GPS and the internet.
Just to name a few.
Even Google got its start with a government grant.
But you won't see that on the first page of search results.
Wow, sounds like public funding is awesome.
Finally a funding source with no downsides.
Well there is one little wrinkle.
Federal science budgets are controlled by politicians.
I hereby award you, Winnie, this government grant for (WHISPERING) What's that, Senator? You'd rather have tax cuts than scientific knowledge? And there's a powerful lobby that won't like the research? And you don't know enough about science to care about the consequences? (SIGHS) Sorry, I'm out.
No! This work is worth the money.
Ask any scientist, they'll tell you.
Sorry, politicians tell me what to do, not scientists.
Maybe not the best system.
But science needs the money.
Yes, it does.
Private funding will never be enough to support research in the public interest.
The kind of long-term careful science that doesn't generate profits or glory, but improves people's lives and increases our understanding of the universe.
We need public funding for science.
But in the past 50 years, the portion of the discretionary budget that government spends on research and development has been cut nearly in half.
And with less funding, there's more competition for fewer resources.
In the past two decades, 30% of National Health Institute grants got funded.
But today, it's about 17%.
No, no, no! And from 2010 to 2013, the U.
S.
had the biggest three-year reduction of scientific research and development since the end of the space race.
(SIGHS) But if it's so hard to get funding, then how do we do science? Well, sometimes we just don't.
In a recent study of 11,000 scientists, half of them said they've had to stop doing valuable research partway through because of economic pressures.
Who knows what great advancements we lost out on as a result.
As much as we'd love to believe that the progress of science is determined solely by the best and brightest, decisions about which science gets funded have always been controlled by non-scientists.
It's all about who holds the purse strings.
That may be true but science always triumphs in the end.
Look at all these journals.
Great, factual peer reviewed research is published every day.
(SUCKING TEETH) Ah - Ugh.
- What? Nothing, just that a surprising number of scientific studies do not hold up.
(SCREAMING) - Oh! - Excuse me.
I'm using those studies for my project.
Go ahead, but there's a good chance a lot of those studies are wrong.
Adam, please, when a study has been published, that means its data is robust and accurate.
Winnie, if you'd ever read "US Weekly", you'd know that just 'cause something is published, that doesn't mean it's true.
What's "US Weekly?" Wow, you are nerdier than I am.
(CHUCKLES) Studies can be flawed for a number of reasons.
If the sample size was too small, the luck of the draw could've skewed the results.
We should test this on 10,000 people but I only know ten.
That's okay, making friends is hard.
(ADAM) Or maybe there was a confounding variable.
Like, if the subject was using another product that could affect the results and the researchers didn't account for it.
Oh, wow, this pill is really boosting our subject's resistance to sunburn.
(ADAM) And something, very rarely, unscrupulous researchers will fake their results.
What if we just make up numbers that show that this pill cures cancer? Hello? Nobel Prize committee? I've got some data you've gotta see.
Sure, so we just need a way to find out which studies are accurate.
We already have one.
And that's to reproduce them.
To show the results weren't just some fluke, someone else has to do another study that tests the same question to see if they can get the same findings.
It's like double checking the answers.
Curse, my results were way different.
And here's the problem.
Since studies are usually reproduced after they're published, that means they have a chance to influence other research or even become famous before we find out they're wrong.
For instance, remember that famous study on power posing.
Uh, it's only one of the most watched TED Talks of all time.
It proved that by standing in the power pose, your body produces hormones that give you confidence.
Well, no one's ever been able to reproduce those results.
What? Or how about that series of studies from the '70s and '80s that showed you can actually make yourself feel happier just by smiling? Well, that makes sense, right? Sure, except that in 2016, almost 20 labs tried to replicate it and not one of them was able to reproduce those findings.
(TRAIN WHISTLE BLOWS) Holy Hawking! If that's true, then how many other studies might be wrong? Oh, a lot of them.
In fact, many scientists say that we're in the midst of a reproducibility crisis.
Like a good type of crisis, right? One research team just replicated 100 famous psychology studies but found they couldn't reproduce about 60% of them.
(BEEP) Another revisited 67 major drug studies and found that about 75% didn't match their results.
And another team zeroed on 53 recent cancer studies and couldn't reproduce 47 of them.
(BEEP, LOUD EXPLOSION) But those numbers are so high! If science gets it wrong that often, then we should be reproducing every study.
(MAN) That'd be nice, but it probably won't happen.
There and that's how you fold a fitted sheet.
This is Professor Nosek.
of the University of Virginia.
And the co-founder of the Center for Open Science.
Adam's right, Winnie.
Replication studies are not getting done.
And it's because scientists don't have enough good reasons to do them.
Wait, what do you mean? To succeed in their careers, scientists have to publish and get attention for their research.
And that's more likely to happen if their studies have new and flashy conclusions.
My study found that Fidget Spinners give you six-pack abs.
(CROWD GASPS AND APPLAUDS) And the pressure to get these kinds of results sometimes leads researchers to cut corners.
I didn't need test subjects.
I just asked my bro, Kevin.
He's super ripped.
On the other hand, replication studies make for good science.
But they don't get as much attention.
People just aren't interested in studies that don't find anything new.
And I replicated a 2003 study on classification of fine-grained polypeptide folds.
The results held up.
(CRICKETS CHIRPING, MAN COUGHS) Hey, her work is just as important.
Yeah, but it isn't as exciting.
Science is competitive.
There aren't enough jobs or grant dollars for everyone.
So, if you were trying to get people excited about your research, what would you do? Follow-up on someone's prior findings? Or try to find something new and amazing? I, uh, also did a study that found candy cures leukemia! (CROWD GASPS AND APPLAUDS) (SHUTTERS CLICKING) These incentives to publish lead to a lot of unreliable research.
One estimate suggest that in the U.
S.
, we spend $28 billion a year on biomedical research that can't be reproduced.
That's such a waste! Well, flashy findings can be useful for stimulating new areas of discovery.
But we need to change the incentives so that researchers are rewarded for producing results that are interesting and reproducible.
Thanks, Professor Nosek.
No problem.
Now if you'll excuse me, I'm gonna go use the emergency shower.
I save a ton on my water bill.
Oh, thrifty.
(CHUCKLING) So, even if my research gets positive results, I still have to find someone to reproduce it, but no one will because they only care about their own careers? Wait, what are you doing? You're right, Adam.
Science sucks, I'm done.
I'm gonna go drink some Red Bull in the woods or do whatever cool kids do.
Oh, wait! I was about to tell you why science is actually amazing! I thought the structure of my show! (BIRDS CHIRPING) You forgot this.
I don't need it.
I'm quitting the competition.
You know, I used to love that no matter how chaotic and uncertain the world felt, I could at least count on science to always be right.
But you showed me, it's not.
Winnie, do you know why science is so awesome? It's the field of human inquiry that embraces mistakes and learns from them.
Scientists don't just assert what they think is true.
They test their hypotheses against the empirical evidence.
And if it turns out they're wrong, they admit it and try again.
The fact that science seeks out its own errors and corrects them is its greatest strength.
But how do we fix what's wrong with science itself? With more science.
With more science.
When I learned about the reproducibility challenges in science, I thought we need to study this problem.
So we started The Center for Open Science to investigate why is it that researchers are not producing reproducible science even though they want to? So, we built software so that researchers can share their data more easily and be more transparent in their methods.
We also work with journals and societies and funders to align the incentives so that it's in researchers interests to be more open and reproducible in their work.
And when I first emigrated here from Pakistan, I had no idea how much research in the U.
S.
rested on flawed mouse testing.
To combat this problem, I started to collect human tissue, and this is how I found a novel treatment for pre-leukemia.
- Whoa.
- So, if something in science is broken like mouse models or faulty studies, it doesn't mean we failed.
It means we need people like you to find the solution by using Science! (ALL) Yes! You're right! I have to use science to fix science.
Eureka! Thank you, guys.
(FAINT CHATTER) And finally Winnie.
And your project is Wait, where is your project? Dr.
Sherman, I didn't do a project.
I did 19.
Because for my experiment, I replicated everyone else's research, and I found that all of their conclusions were irreproducible.
Wow, well, that doesn't quite follow the project guidelines.
But, nice work, Winnie.
Sorry, everyone else.
(MUFFLED GRUMBLING) You suck.
Well, I didn't do it for the glory.
I did it for science.
Come on, Adam, I'll show you my new research proposal.
I got some sweet new binder tabs.
Ooh, I love office supplies.
(CHUCKLING) (BOY GASPS) The card moved.
Did anyone else see that? Holy (SCHOOL BELL RINGS) Hello, we're here on set with Brian Nosek because I have way more questions for him about the reproducibility crisis in science.
Thanks for having me.
Is the crisis real and is it something that we should all be worried about? Well, it's at least a crisis of confidence.
And it's, uh, something that emerged when researchers across a variety of disciplines realized that the published literature, what we thing we're doing and producing and finding isn't as reliable as we think it is.
And so, can you walk us through a little bit of how the fields realized this was happening.
Well, the oddity is is that we've known the challenges for reproducibility for 60 years.
And it hasn't gotten a groundswell.
But that has shifted over the last ten years because perhaps the internet helped make it easier to evaluate lots of different papers at the same time.
Perhaps because of a couple of very dramatic findings that there has been a convergence.
And so it has become a thing.
Now that we know about it, we can do something about it.
That's exactly right.
Science is at its best when it is constantly self critiquing of looking at where it's limited, where it's not finding things that are reliable and looking for ways to do it better.
Every scientific claim is wrong in some fundamental way.
And the process of discovery in science is figuring out where our current models of the world are wrong and how we can make them less wrong.
I'm so fascinated by that.
Can you elaborate on every scientific theory about the world is wrong in some way.
In some way.
If we had a perfect model of the world, it would be as complex as the world.
How it is we describe how the world works, how people interact, how atoms interact, whatever it is, is a simplification.
- It is a way to try to describe - Right.
lots of different things that are happening in simple terms that make predictions about next things that will happen.
Right.
But our predictions are always imperfect.
They can't quite get exactly the phenomena as we study them.
So, they're wrong.
Right.
But wrong doesn't mean, oh, that's useless.
It's not information that we need.
It's not information that we can use.
Newton's Laws of Mechanics are the most successful theory ever, right? We use them to build bridges and skyscrapers I learned about them in high school, yeah.
And they're wrong, right? Einstein showed how they are wrong when things are moving very fast.
So, it isn't that something being wrong means, oh, can ignore science.
It means there's always something more to learn.
So, that leads me really nicely to my next question which is Is this a reason that people should stop believing or trusting science? You know, should the audience go home and every time they read about a new study say, oh, that's probably never gonna be reproduced and so I'm not even gonna think about it again? One of the big challenges of scientific knowledge is that is accumulates very slowly.
So, no one study is definitive.
Drinking coffee is bad for you.
No, drinking coffee is good for you.
No, drinking coffee's bad for you.
Well, make up your mind, scientists.
It's those claims that are initially very exciting.
But then survive confrontation over time are the ones that survive and we have confidence in.
So, as a reader of the newspaper of reporting scientific results, one thing to look for is, is this the first finding? Is this something brand-new? Uh, or is it the accumulation of a lot of different pieces of evidence? That's a very easy signal for whether that's a credible claim or a reliable claim that we know a lot about it versus something new and possibly exciting.
Yeah.
The reason to trust science is because it doesn't trust itself.
Ahh.
Right? It's that science is always questioning its own findings which makes it the most reliable way to discover things.
Thank you so much for being here.
Thanks for having me, Adam.