Early clinical trials not always reliable, Mayo researchers find

If the data supporting a new medical treatment look too good to be true, they just might be.

Researchers at the Mayo Clinic recently examined more than 900 randomized controlled trials published in top-shelf medical journals in the past decade and found that a surprising number of the earliest studies on treatments for chronic conditions presented exaggerated results.

"This phenomenon is termed the 'Proteus effect,' " Dr. Alex Krist, a Virginia physician and vice chairman of the U.S. Preventive Services Task Force, wrote in an editorial about the Mayo study of studies. "It means that knowing the answer to a question involves a body of evidence, not a definitive trial. Early trials should be viewed with caution."

The findings could have implications for everything from conversations with doctors about new treatments to product approvals and insurance decisions. The Food and Drug Administration, which often approves products based on early studies, is reviewing the paper's findings.

To look for the Proteus effect, researchers with Mayo's Center for the Science of Health Care Delivery examined 70 different meta-analyses of 930 individual clinical trials. The studies examined drug and medical device treatments for common chronic conditions that drive health care spending in aging populations like diabetes, stroke, chronic obstructive lung disease (COPD), coronary artery disease and kidney disease.

All of the meta-analyses of past studies were published in well-known journals like JAMA, the Lancet, and the New England Journal of Medicine, as well as Mayo Clinic Proceedings, which published the Proteus-effect paper in its March issue. The Mayo team found that in more than a third of the meta-analyses they examined — 37 percent, specifically — the first two studies on a given treatment reported a much greater treatment effect than later studies.

On average, these early studies reported treatment effects that were more than 2.5 times greater than the overall results for the same intervention. In the four most egregious examples, the initial treatment effects were five to nine times greater than the overall effects documented in the meta-analyses, according to the study's corresponding author, Mayo epidemiologist Dr. M. Hassan Murad.

"Usually, we are waiting for new innovations, new interventions or new tests, with hope. We want something to be effective," Murad said in an interview. "But unfortunately, it seems that what we see in the first or second study [sometimes] is not as impressive as what we see later when the studies are repeated by other people in different settings with different patients."

One of the meta-analyses that documented the greatest Proteus effect, according to Murad, was a review of 24 past clinical trials on whether long-term maintenance therapy with antipsychotic drugs can prevent relapses of future schizophrenic episodes.

The review, published in the Lancet in June 2012, found 27 percent of patients who took antipsychotics experienced relapses between seven and 12 months after their initial episode, compared to 64 percent of the control patients who had relapses in that time while on a placebo. Although the initial trials reached the same positive conclusions as later trials, the effect was more pronounced in the early work than was seen over time in later studies, the Mayo study found.

The Lancet paper's authors themselves noted the "heterogeneous" results they found at the time, and said the differences were matters of degree, not "direction of effect." For example, although the antipsychotic drugs reduced the risk of relapse in all patients, it was more robust in specific subgroups, like people who had only one episode and those in remission.

A key difference between early clinical trials and those that come later is the study population: Early studies are often run on carefully groomed populations that are likely to see the greatest effects, and thus produce the most positive data, while later studies tend to examine larger and more diverse groups with confounding factors like multiple overlapping health problems and widely varying ages.

The Mayo team examined 16 different factors that could be linked to skewed early research, like study duration, selective risk reporting, and funding source. Although having a single study site instead of multiple sites was the most predictive factor for the Proteus effect, in truth none of the factors including that one was strongly correlated to the Proteus effect. "At least for now, the Proteus effect is unpredictable," the report concludes.

The Mayo paper adds to a growing body of evidence that casts skeptical light on early medical research, but Murad said he doesn't view it as "anti-innovation" and would not want to see the results used by insurers to deny treatments.

"We're not saying they [doctors] shouldn't give the medicine until they have a lot of studies," Murad said. "We're saying, you do give the treatment if the patient agrees to it, but you will be skeptical, and the patient can be a little skeptical. That's OK."