« What happens when information disappears. | Main | Old technologies »
September 6, 2005
Quality of medical science information
Science is the process of continuing discovery. Part of that process of discovery is testing of hypothesis - normally using statistical methods. However, those tests are not always accurate. A story in the Economist, Scientific accuracy ... and statistics , describes a recently published report by Dr. John Ioannidis, a Greek epidemiologist on quality of medical studies:
One [well-know source of error] is an unsophisticated reliance on “statistical significance”. To qualify as statistically significant a result has, by convention, to have odds longer than one in 20 of being the result of chance. But, as Dr Ioannidis points out, adhering to this standard means that simply examining 20 different hypotheses at random is likely to give you one statistically significant result. In fields where thousands of possibilities have to be examined, such as the search for genes that contribute to a particular disease, many seemingly meaningful results are bound to be wrong just by chance.
Other factors that contribute to false results are small sample sizes, studies that show weak effects (such as a drug which works only on a small number of patients) and poorly designed studies that allow the researchers to fish among their data until they find some kind of effect, regardless of what they started out trying to prove. Researcher bias, due either to clinging tenaciously to a pet theory, or to financial interests, can also skew results.
When Dr Ioannidis ran the numbers through his model, he concluded that even a large, well-designed study with little researcher bias has only an 85% chance of being right. An underpowered, poorly performed drug trial with researcher bias has but a 17% chance of producing true conclusions. Overall, more than half of all published research is probably wrong.
That is a very large error rate - and cries out for better quality control.
Part of the problem maybe our continuing lack of bio-medical knowledge. The human body is such a complexity that we are, in many ways, still guessing about how it operates at the micro level. As the Economist points out:
The physical sciences, with more certain theoretical foundations and well-defined methods and endpoints, probably do better than medicine.
Even so, it would be well to remember that the scientific models we create are approximations of the real world - not the real world itself. We are still, ultimately, guessing. And the development of data and information is an ongoing activity - not a static event:
His [Dr. Ioannidis'] thesis that many scientific papers come to false conclusions is not new. Science is a Darwinian process that proceeds as much by refutation as by publication.
The best we can do is continue the research and testing - with full knowledge that what ever we come up with maybe wrong and/or in need of further refinement.
Posted by Ken Jarboe at September 6, 2005 9:34 AM
Trackback Pings
TrackBack URL for this entry:
http://www.athenaalliance.org/mt/mt-tb.cgi/311