So What Do The Terms "Effective" or "Significant" Mean To You?

A recent headline reporting the results of a study on an experimental Johnson & Johnson diabetes drug read as follows:

J&J diabetes drug effective in high risk patients - study - Reuters

So. What would effective mean to you?




I can tell you what it WOULDN'T mean to me.

It wouldn't mean a less than impressive .65-.73% change in A1C levels after 18 weeks, as was the case in this particular study. But according to one of the lead researchers, David Matthews, "These are clinically significant drops, no doubt about it," .

This spawns a new question - What exactly does "clinically significant" mean? Besides the obvious sad answer of less than 1%seen here, the following video gives us the even sadder definition:

"...95% certain results not due to chance."

Well, you must feel better now knowing the most decisions and recommendations many doctors make regarding which drugs you should take are due to this wonderful standard right?

Yeah, me neither.

The researchers, however, were even further thrilled that this new drug also led to "statistically significant reductions" in systolic blood pressure. How much you ask?

a 2.6 mmHg-4.4mmHg drop

 

Does anyone here really think a blood pressure drop of 140/80 to 136/80 means anything?

 

But again, according to Matthews, "Blood pressure certainly comes down in significant amounts, in fact in amounts that you would regard to be quite respectable in an anti hypertension agent,".

 

If you knew that was about all the change you could expect by taking this drug, paying it's co-pay, and dealing with it's side effects - would you still bother taking it?

 

Are you upset you weren't told these less than impressive results were about the best you could expect or hope for?

 

The video above also details how statistics are fudged when it comes to the relationship between salt restriction and blood pressure, as well as statin use and heart attack prevention.

 

So why the sensational headlines?

 

The following passage from THIS article from The Atlantic sheds some light:

"To get funding and tenured positions, and often merely to stay afloat, researchers have to get their work published in well-regarded journals, where rejection rates can climb above 90 percent. Not surprisingly, the studies that tend to make the grade are those with eye-catching findings."

 So remember, while to normal people like ourselves "significant" typically means something like "impressive, meaningful, or important", to a researcher it simply means:

 

"...95% certain results not due to chance."

 

About the only thing "significant" found is this study to me was the most common side effect....

 

genital infections.