One problem with medical journals is that much of what they contain is
plain wrong. And much of what's not wrong simply doesn't matter. Canadian
researchers have for years been combing medical journals to find research
articles that matter to patients and are "true" in that their conclusions
are supported by their methods and data: they find that it's less than 5%
of the article and for most journals less than 1%.

More troublesome is the bias in the research. The highest form of
evidence in medicine is the randomised controlled trial, and about two
thirds of the trials in major journals are funded by the pharmaceutical
industry. Unfortunately--and stupidly I never realised this until I
retreated to a 15th century palazzo in Venice to write a book on medical
journals--those trials almost always come up with results favourable to the
company. It's not that they fiddle the results, but they are clever with
the questions they ask and the methods they use. They are so clever that
every study funded by the drug industry of whether newer contraceptive
pills caused more blood clots found that they didn't, whereas every study
funded with public money found they did. I was led to the reluctant
conclusion that medical journals are in some ways extensions of the
marketing arm of drug companies.

(You might have spotted by now that this blog is partly a promotion of
my book "The trouble with medical journals." Evidently such things are
acceptable in the blogosphere. It isn't much of a book, but it might be
worth a glance.)

One reason that journals publish so much rubbish is because their
method of assuring quality--peer review--is hopeless. Broadly peer review
the process whereby one or more peers of the authors of a study pass a
judgement on the study, usually anonymously. It lies at the heart of
science and determines which research get funded, which studies are
published, who is promoted, and who wins a Nobel prize. Despite being
central to science it had never until 20 years ago been studied, a paradox
for a way of studying the world that depends on experimentation and data.
When the studies began they showed that peer review was slow, expensive,
largely a lottery, ineffective at detecting error, prone to bias, easily
abused, and entirely useless for picking up fraud. In one study of peer
review the researchers inserted eight errors in a 600 word article and sent
it to 400 reviewers: the median numbers of errors that the reviewers
spotted was two; nobody spotted more than five; and a fifth of the
reviewers didn't spot any. As the now famous saying goes: "If peer review
was a drug it wouldn't be allowed on the market." Yet it continues to be a
sacred belief for an intellectual discipline that scorns faith and demands
evidence.

In my book I explore many other defects in medical journals, but
perhaps the most disturbing is the publication of fraudulent studies.
Nobody knows how many studies are fraudulent, but we now have a series of
cases stretching back 50 years. We were shocked by the first cases and
assumed that they must be rare and due to mental problems among
researchers. This was naïve. All human activity is associated with fraud,
and in science it is easy to commit fraud because the system depends on
trust. If authors say that they studied 200 patients they are believed:
nobody asks for pictures, signatures, or medical records.
As an editor I came across many instances of fraud--and particularly
disturbing were three cases where authors had probably published dozens of
fraudulent studies in prominent journals but where nobody has investigated
the studies. We are left not knowing whether they are fraudulent or
not--and so whether to use or discard them. The major scandal is not that
fraud happens but that science does not have adequate methods of preventing
and managing the problem.
I grow boring and "old mannish," and maybe my book is boring. But it is, I
contend, important for us all to understand the many problems with medical
journals--otherwise, our debates on health, sickness, birth, and death may
be corrupted from the beginning.