dotlah! dotlah!
  • Cities
  • Technology
  • Business
  • Politics
  • Society
  • Science
  • About
Social Links
  • zedreviews.com
  • citi.io
  • aster.cloud
  • liwaiwai.com
  • guzz.co.uk
  • atinatin.com
0 Likes
0 Followers
0 Subscribers
dotlah!
  • Cities
  • Technology
  • Business
  • Politics
  • Society
  • Science
  • About
  • Science
  • Technology

One Reason So Many Scientific Studies May Be Wrong

  • November 25, 2019
Total
0
Shares
0
0
0

There is a replicability crisis in science – unidentified “false positives” are pervading even our top research journals.

Statistics: if you torture the data enough, they will confess. clemsonunivlibrary/ Flickr, CC BY-NC

A false positive is a claim that an effect exists when in actuality it doesn’t. No one knows what proportion of published papers contain such incorrect or overstated results, but there are signs that the proportion is not small.

The epidemiologist John Ioannidis gave the best explanation for this phenomenon in a famous paper in 2005, provocatively titled “Why most published research results are false”. One of the reasons Ioannidis gave for so many false results has come to be called “p hacking”, which arises from the pressure researchers feel to achieve statistical significance.

What is statistical significance?

To draw conclusions from data, researchers usually rely on significance testing. In simple terms, this means calculating the “p value”, which is the probability of results like ours if there really is no effect. If the p value is sufficiently small, the result is declared to be statistically significant.

Traditionally, a p value of less than .05 is the criterion for significance. If you report a p<.05, readers are likely to believe you have found a real effect. Perhaps, however, there is actually no effect and you have reported a false positive.

Many journals will only publish studies that can report one or more statistically significant effects. Graduate students quickly learn that achieving the mythical p<.05 is the key to progress, obtaining a PhD and the ultimate goal of achieving publication in a good journal.

This pressure to achieve p<.05 leads to researchers cutting corners, knowingly or unknowingly, for example by p hacking.

The lure of p hacking

To illustrate p hacking, here is a hypothetical example.

Bruce has recently completed a PhD and has landed a prestigious grant to join one of the top research teams in his field. His first experiment doesn’t work out well, but Bruce quickly refines the procedures and runs a second study. This looks more promising, but still doesn’t give a p value of less than .05.

Convinced that he is onto something, Bruce gathers more data. He decides to drop a few of the results, which looked clearly way off.

He then notices that one of his measures gives a clearer picture, so he focuses on that. A few more tweaks and Bruce finally identifies a slightly surprising but really interesting effect that achieves p<.05. He carefully writes up his study and submits it to a good journal, which accepts his report for publication.

Bruce tried so hard to find the effect that he knew was lurking somewhere. He was also feeling the pressure to hit p<.05 so he could declare statistical significance, publish his finding and taste sweet success.

There is only one catch: there was actually no effect. Despite the statistically significant result, Bruce has published a false positive.

Bruce felt he was using his scientific insight to reveal the lurking effect as he took various steps after starting his study:

  • He collected further data.
  • He dropped some data that seemed aberrant.
  • He dropped some of his measures and focused on the most promising.
  • He analysed the data a little differently and made a few further tweaks.

The trouble is that all these choices were made after seeing the data. Bruce may, unconsciously, have been cherrypicking – selecting and tweaking until he obtained the elusive p<.05. Even when there is no effect, such selecting and tweaking might easily find something in the data for which p<.05.

Statisticians have a saying: if you torture the data enough, they will confess. Choices and tweaks made after seeing the data are questionable research practices. Using these, deliberately or not, to achieve the right statistical result is p hacking, which is one important reason that published, statistically significant results may be false positives.

What proportion of published results are wrong?

This is a good question, and a fiendishly tricky one. No one knows the answer, which is likely to be different in different research fields.

A large and impressive effort to answer the question for social and cognitive psychology was published in 2015. Led by Brian Nosek and his colleagues at the Center for Open Science, the Replicability Project: Psychology (RP:P) had 100 research groups around the world each carry out a careful replication of one of 100 published results. Overall, roughly 40 replicated fairly well, whereas in around 60 cases the replication studies obtained smaller or much smaller effects.

The 100 RP:P replication studies reported effects that were, on average, just half the size of the effects reported by the original studies. The carefully conducted replications are probably giving more accurate estimates than the possibly p hacked original studies, so we could conclude that the original studies overestimated true effects by, on average, a factor of two. That’s alarming!

How to avoid p hacking

The best way to avoid p hacking is to avoid making any selection or tweaks after seeing the data. In other words, avoid questionable research practices. In most cases, the best way to do this is to use preregistration.

Preregistration requires that you prepare in advance a detailed research plan, including the statistical analysis to be applied to the data. Then you preregister the plan, with date stamp, at the Open Science Framework or some other online registry.

Then carry out the study, analyse the data in accordance with the plan, and report the results, whatever they are. Readers can check the preregistered plan and thus be confident that the analysis was specified in advance, and not p hacked. Preregistration is a challenging new idea for many researchers, but likely to be the way of the future.

Estimation rather than p values

The temptation to p hack is one of the big disadvantages of relying on p values. Another is that the p<.05 criterion encourages black-and-white thinking: an effect is either statistically significant or it isn’t, which sounds rather like saying an effect exists or it doesn’t.

But the world is not black and white. To recognise the numerous shades of grey it’s much better to use estimation rather than p values. The aim with estimation is to estimate the size of an effect – which may be small or large, zero, or even negative. In terms of estimation, a false positive result is an estimate that’s larger or much larger than the true value of an effect.

Let’s take a hypothetical study on the impact of therapy. The study might, for example, estimate that therapy gives, on average, a 7-point decrease in anxiety. Suppose we calculate from our data a confidence interval – a range of uncertainty either side of our best estimate – of [4, 10]. This tells us that our estimate of 7 is, most likely, within about 3 points on the anxiety scale of the true effect – the true average amount of benefit of the therapy.

In other words, the confidence interval indicates how precise our estimate is. Knowing such an estimate and its confidence interval is much more informative than any p value.

I refer to estimation as one of the “new statistics”. The techniques themselves are not new, but using them as the main way to draw conclusions from data would for many researchers be new, and a big step forward. It would also help avoid the distortions caused by p hacking.The Conversation

 

Geoff Cumming, Emeritus Professor, La Trobe University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Total
0
Shares
Share
Tweet
Share
Share
Related Topics
  • Journals
  • Open Science
  • Research
  • Scientific Publication
  • Statistics
majulah

Previous Article
  • Technology

Best Deals On Earphones You Can Get On AliExpress Today

  • November 25, 2019
View Post
Next Article
  • Technology

Singapore Signs MOU With The Republic Of Korea To Enhance Cybersecurity Cooperation

  • November 25, 2019
View Post
You May Also Like
View Post
  • Gears
  • Technology

Meet Samsung Galaxy Tab S11 Series: Packing Everything You Expect From a Premium Tablet

  • Dean Marc
  • September 4, 2025
View Post
  • Technology

Malaysia’s ‘ASEAN Shenzhen’ needs some significant legal reform to take off — here’s how

  • dotlah.com
  • August 25, 2025
View Post
  • Gears
  • Technology

Samsung Electronics Debuts Odyssey G7 Monitors, Showcasing Top Games on Its Displays at Gamescom 2025

  • Dean Marc
  • August 20, 2025
View Post
  • Artificial Intelligence
  • Technology

Thoughts on America’s AI Action Plan

  • Dean Marc
  • July 24, 2025
View Post
  • Technology

ESWIN Computing launches the EBC77 Series Single Board Computer with Ubuntu

  • dotlah.com
  • July 17, 2025
View Post
  • Gears
  • Technology

Samsung Galaxy Z Fold7: Raising the Bar for Smartphones

  • Dean Marc
  • July 9, 2025
View Post
  • Cities
  • Technology

Meralco PowerGen’s PacificLight starts up 100 MW fast-response plant in Singapore

  • dotlah.com
  • June 20, 2025
View Post
  • Technology

Apple services deliver powerful features and intelligent updates to users this autumn

  • Dean Marc
  • June 12, 2025


Trending
  • automobile-graffiti-4030200_1280 1
    • Cities
    How To Avoid Pedestrian Accidents And Stay Safe On The Road
    • October 15, 2021
  • 2
    • Lah!
    Too Good To Be True: Let’s Fight Scams
    • May 3, 2019
  • host-countries-of-afghan-refugees-25559 3
    • People
    Where Afghan Refugees Are Located
    • August 19, 2021
  • driver-car-dog-cdc-A82PSKGx9cI-unsplash 4
    • Cities
    4 Tips To Help You Prepare For Your Driver’s License Test
    • October 30, 2021
  • 5
    • Artificial Intelligence
    • People
    • Technology
    Usher’s New Look Embarks on AI-Focused Collaboration with IBM to Set Students up for Career Success
    • March 7, 2024
  • 6
    • Technology
    Making The Quantum Leap In Computing
    • February 4, 2020
  • factory-worker-5736096_1280 7
    • Business
    • People
    Understanding the Purpose of Reasonable Suspicion Training
    • November 23, 2021
  • 8
    • Technology
    NTU Singapore And Google Cloud Develop New Rapid-response Virtual Assistant To Help Address Freshmen Queries
    • July 24, 2020
  • 9
    • Science
    Physicists Design An Experiment To Pin Down The Origin Of The Elements
    • August 30, 2019
  • 10
    • Lah!
    Maritime And Port Authority Of Singapore (MPA) To Strengthen Collaboration On Cyber Resilience And Response
    • October 16, 2019
  • 11
    • People
    COVID-19: It’s Time To Rethink Where, How And Why We Work
    • September 1, 2020
  • 12
    • Lah!
    • Society
    IPOS Director Named As Influential Women In IP 2020 BY World Intellectual Property Review
    • June 18, 2020
Trending
  • Kitchen Knives - Best All-Around Picks in 2025 1
    Kitchen Knives : The Surgeons of Cooking – Best All-Around Picks in 2025
    • September 15, 2025
  • 2
    Apple unveils iPhone 17 Pro and iPhone 17 Pro Max, the most powerful and advanced Pro models ever
    • September 9, 2025
  • 3
    Apple debuts iPhone 17
    • September 9, 2025
  • Fruits and vegetables for cooking. For food processors. 4
    Food Processor: The Swiss Army Knife of the Kitchen – Best All-Around Picks in 2025
    • September 8, 2025
  • 5
    Meet Samsung Galaxy Tab S11 Series: Packing Everything You Expect From a Premium Tablet
    • September 4, 2025
  • 6
    Malaysia’s ‘ASEAN Shenzhen’ needs some significant legal reform to take off — here’s how
    • August 25, 2025
  • French Fries 7
    Air Fryer: The One Cooking Appliance to Rule Them All – Best All-Around Picks in 2025
    • August 22, 2025
  • 8
    Samsung Electronics Debuts Odyssey G7 Monitors, Showcasing Top Games on Its Displays at Gamescom 2025
    • August 20, 2025
  • 9
    HP Cranks Up the Game with Smarter Systems, Cooler Builds, and Gear That Hits Different
    • August 14, 2025
  • 10
    New Trump tariffs: early modelling shows most economies lose – the US more than many
    • August 6, 2025
Social Links
dotlah! dotlah!
  • Cities
  • Technology
  • Business
  • Politics
  • Society
  • Science
  • About
Connecting Dots Across Asia's Tech and Urban Landscape

Input your search keywords and press Enter.