Reproducibility

I touched on this in one of the climate threads, according to this article scientists are having significant difficulties reproducing each other's results. The tendency is to promote "big results" so that journals sell and funding bodies justify themselves. The problem is that if results aren't reproducible then it ain't science.

http://www.bbc.co.uk/news/science-environment-39054778

Originally posted by DeepThought
I touched on this in one of the climate threads, according to this article scientists are having significant difficulties reproducing each other's results. The tendency is to promote "big results" so that journals sell and funding bodies justify themselves. The problem is that if results aren't reproducible then it ain't science.

http://www.bbc.co.uk/news/science-environment-39054778

Science is hard.

I have seen studies published recently which divided a mutant mouse colony in two, and had two different groups perform the same experiments simultaneously. Remarkably, the results were different. In both cases, there was a clear observed effect. It was just a different effect between the two facilities (I wish I had the link but can't dig it up.)

We can't know all the variables. Studies being conducted today are collaborative, highly rigorous, and almost always ethically sound experimentally. The problem of reproducibility is real and important, but not overly concerning to practicing scientists. Given the sophistication of methods, a real discovery requires a scientific consensus. There needs to be a series of publications, from multiple labs, that confirm the same conclusion in multiple systems.

There are many examples of scientific "discoveries" that were never followed up on in the literature. If you're a scientist going through this literature, you conclude that the results could not be reproduced. The findings just end up in the garbage dump of bad scientific conclusions.

Originally posted by DeepThought
I touched on this in one of the climate threads, according to this article scientists are having significant difficulties reproducing each other's results. The tendency is to promote "big results" so that journals sell and funding bodies justify themselves. The problem is that if results aren't reproducible then it ain't science.

I have also talked about such problems before. But we must be careful. Science could be said to be the process of doing the studies, doing the reproducing etc. So to simply say 'it isn't science' because someone couldn't reproduce it is a bit simplistic. Someone needs to do the first experiment. That is part of the scientific process. The conclusions should not be taken as fact until the experiment has been reproduced. But failure to reproduce does not necessarily mean the first experimenter wasn't doing science.
Probably the biggest problem here is not so much the scientists or their process, but the media and the way they report science, often assuming it is proven fact when the first study results come out. Also scientists have a tendency to speculate about conclusions (that don't necessarily follow from the results) but they qualify such speculation. Then reporters pick it up, strip out the qualifications and report it like it was a direct conclusion from the study - when it wasn't.

There are many other problems with the scientific community including how scientists are recognised, how funding is obtained, how information is shared - or not shared etc.

This is worth watching on the subject:
YouTube&t=2s[/youtube]

Originally posted by twhitehead

The conclusions should not be taken as fact until the experiment has been reproduced.

Even that is too simplistic because it depends on the credibility that the conclusion of the first experiment could be wrong. If there is no credibility that the conclusion of the first experiment could be wrong, then it is scientific fact without reproducibility.

An example of that is the first successful experiment in powered aircraft with the (trivial) conclusion that it is possible to fly with powered aircraft; (not that scientists generally said it was impossible before that or at least not that I am aware of) that was rightly considered scientific fact before it was sequentially reproduced as it was rightly not considered creditable from the first trivial observation (of actual flight) that that conclusion could be wrong.

In my book I am writing to be published, I will explain the true essence of science and explain all the common myths of what science is.
I will assert that the true essence of science isn't about reproducibility nor about producing falsifiable theories nor testable theories because, although each one of those things definitely helps and make it much more likely to be valid science, what the true essence of science is nothing more than totally rationally producing theories i.e. base the theories PURELY on the evidence and flawless logic (or just flawless logic in the case of maths theorem) and nothing else (such as flawed logic or emotions or baseless superstition).

incidentally, I really don't like the way wiki explains reproducibility;
https://en.wikipedia.org/wiki/Reproducibility
"..Reproducibility is the ability of an entire analysis of an experiment or study to be duplicated, ..."
What if that analyses uses illogic? It still can be duplicated! Just not rationally.
ANY "entire analysis of an experiment" can be duplicated, no matter HOW stupid or illogical it is!
No, reproducibility is not the ability of an "entire analysis of an experiment" to be duplicated but rather the ability of the "experiment and the observed result" to be duplicated. Perhaps someone here should correct the edit of that?

Originally posted by humy
Even that is too simplistic because it depends on the credibility that the conclusion of the first experiment could be wrong. If there is no credibility that the conclusion of the first experiment could be wrong, then it is scientific fact without reproducibility.

Well of course much of science is little more than observation which isn't really experiment at all and often doesn't require reproduction.

Originally posted by humy
In my book I am writing to be published, I will explain the true essence of science and explain all the common myths of what science is.

The danger here is that you are inventing a definition of 'science' then announcing that anyone who uses a different definition is wrong, or talking about a 'myth'.
The reality is the word is actually quite vague and is used in many ways and in many contexts. The 'scientific method' is what we were referring to above.

Originally posted by twhitehead
Well of course much of science is little more than observation which isn't really experiment at all and often doesn't require reproduction.

there is a blurred distinction between an observation and an experiment; many observations, excluding accidental ones, can generally be correctly described as 'experiment' albeit often in a trivial way. An experiment can be defined as deliberate action to observe a result. So the action of going to a telescope and deliberately looking through it to observe whether there is a star in that exact direction can be thought of as a (trivial) 'experiment'. There is no clear cut distinction between such observations and 'experiments' and the distinction between the two tends to be relative and subjective because it generally depends on how 'complex' you say the set of deliberate actions must be before you call it an 'experiment'; most people may think that the mere going to the telescope and looking through it makes the action not 'complex' enough for them to call that an 'experiment' but may be happier to call it an 'experiment' if you had to also design and build the telescope first because that adds much 'complexity' to the total set of deliberate actions before you made your observation.

Originally posted by humy
An experiment can be defined as deliberate action to observe a result.

Yes, I am sure it can. But it is still useful to distinguish between what would normally be called an observation and what would normally be called an experiment. Observations typically do not involve manipulating the target whereas an experiment involves manipulating then observing.
So astronomers generally don't do experiments (by the meaning I use above). It is still often important for multiple groups to doublecheck observations or analysis of observational data.
But even in biology, there is a distinction between observing an animal in the wild and describing its behaviour, and manipulating something in multiple different ways and observing the results with 'control' groups and double blind studies etc. Plain observations in general are less controversial and less likely to require reproduction. Experiments are much more susceptible to bias' of various kinds and very often involve statistical results.
If your powered aircraft only got off the ground 5% of the time, whereas unpowered aircraft that were otherwise identical were getting off the ground 3% of the time, we might reasonably say that it might have been the wind and not the power and the results were inconclusive.

If you can't reproduce it, then it isn't science.

Originally posted by humy
Even that is too simplistic because it depends on the credibility that the conclusion of the first experiment could be wrong. If there is no credibility that the conclusion of the first experiment could be wrong, then it is scientific fact without reproducibility.

An example of that is the first successful experiment in powered aircraft with the (trivial) co ...[text shortened]... and the observed result" to be duplicated. Perhaps someone here should correct the edit of that?

No, with a few caveats, it has to be reproducible. Otherwise we'd all believe in cold fusion. Straightforwardly, if an experimental result cannot be reproduced then there is considerable doubt as to the veracity of the original result. This is different from unpublishable, the point of publishing the experimental result is partly to disseminate the result, but also to allow other groups to check it. I think that the complaint in the article I referenced is that groups are not giving enough detail about their experiments for them to be reproduced. I'd be interested to know which fields this applies to. One would expect more reproducibility if it's happening in physics, where experiments are easier to reproduce than in biological sciences since the subject of the experiment is to all intents and purposes the same in each physics experiment, whereas each laboratory rat is a different individual.

With something like an astronomical observation of a rare event one might make an exception since the event is rare. In that case one can look at the quality of the rest of the work the group has produced and put the scientific equivalent of a health warning on the packet. Then it's a matter of waiting until the event recurs.

A nice example is the detection of gravitational waves at LIGO, they confirm their own results by having two detectors. Virgo, a European collaboration and Indigo (Indian LIGO) will provide confirmation from separate groups. Black hole collisions appear to be relatively commonplace, so the events aren't that rare. They'll probably see a couple of events this year.

Originally posted by DeepThought
No, with a few caveats, it has to be reproducible. Otherwise we'd all believe in cold fusion.

No we wouldn't all believe in cold fusion; cold fusion was NEVER proven experimentally beyond reasonable doubt; not the first time nor at any other time + it was always doubtful from the start because it appears to violate what we know about well established physics.

And any exception to the 'reproducible rule' means it doesn't have to be reproducible to be science.
What if a single one-time event occurred that MANY people currently on Earth clearly observed but it never happened again, such as a magnetic reversal specifically causing all bees to go extinct (unlikely, but that's besides the point) thus proving a magnetic reversal can specifically make all bees go extinct? (with a credible explanation of how so backed up by many observations and no credible alternative explanation or evidence for an alternative cause); this cannot be reproduced in the lab once it has happened because we would then have no bees to experiment with. So we shouldn't believe it to be a scientific fact because it cannot be reproduced?

Originally posted by twhitehead
Observations typically do not involve manipulating the target whereas an experiment involves manipulating then observing..

Point taken. But there is much established scientific fact without such experiments conducted first because much of valid science comes from just observing how things naturally are. What if we all observe a massive close-by supernovae in the sky from a particular star exploding; we cannot ever observe that same star explode ever again i.e. it cannot be later observed to happen to that particular star; so it isn't a scientific fact?
Yes we can observe other stars explode and confirm they can; but what if we hadn't observe any such stars explode before that one; so we have to wait until we observe another star explode before it is fact that the first star exploded? What if the first star exploding was witnessed by millions across the world including all scientists? Surely we would say it is a scientific fact BEFORE any kind of duplication, right?

One can conceive of a hypothetical event that is clearly and irrefutably witnessed by millions, doesn't involve absurdities such as supernatural involvement or magic or violation of laws of physics etc and there is also no other reason to think millions of people were hallucinating or mistaken or are simply lying, and that event then never happens again and cannot be reproduced. How can that still not be a scientific fact at least during the lifetime of those witnesses?
If that hypothetical event would still be a scientific fact then that proves reproducibility is not what defines what is scientific; reproducibility merely helps.

Originally posted by humy
No we wouldn't all believe in cold fusion; cold fusion was NEVER proven experimentally beyond reasonable doubt; not the first time nor at any other time + it was always doubtful from the start because it appears to violate what we know about well established physics.

And any exception to the 'reproducible rule' means it doesn't have to be reproducible to be ...[text shortened]... riment with. So we shouldn't believe it to be a scientific fact because it cannot be reproduced?

You cannot rely on agreement or disagreement with paradigm theories. The paradigm could be wrong. Depending on the paradigm theory it makes it more or less plausible, but not certain. It's not clear to me that cold fusion contradicted established theories, assuming they were using deuterium oxide and not natural water, bubble collapse produces extreme conditions so it's not especially clear that it did contradict established theories.

Multiple groups observing the same event amounts to a reproducible experiment. I have not claimed it did not - my LIGO example explicitly included that.

Originally posted by wildgrass
Science is hard.

I have seen studies published recently which divided a mutant mouse colony in two, and had two different groups perform the same experiments simultaneously. Remarkably, the results were different. In both cases, there was a clear observed effect. It was just a different effect between the two facilities (I wish I had the link but can't ...[text shortened]... d not be reproduced. The findings just end up in the garbage dump of bad scientific conclusions.

I've been trying to think of a way to respond to your excellent post. One thing I've seen in medical research papers are the words "marginal significance". They've done a trial and the drug's failed (one even sees the words "the patient failed the treatment" on occasion) but it's right on the borderline so they call it "marginal significance" when it should be "not significant". I did some editing for the BMJ a while ago which is why I've seen these things. It put me off going to the doctors for life! 😉