From the moment that these registries were first suggested, and then opened, it was implicitly assumed that the shame of producing this public record, and then not publishing your study, would be enough to ensure that people would do the right thing. But the first problem for the US register, which could have been used universally, was that people simply chose not to use it. The regulations required only a very narrow range of trials to be posted, and nobody else was in a hurry to post their trials if they didn’t have to.
In 2004 the International Committee of Medical Journal Editors (ICMJE) – a collection of editors from the most influential journals in the world – published a policy statement, announcing that none of them would publish any clinical trials after 2005, unless they had been properly registered before they began.62 They did this, essentially, to force the hand of the industry and researchers: if a trial has a positive result, then people desperately want to publish it in the most prestigious journal they can find. Although they had no legal force, the journal editors did have the thing that companies and researchers wanted most: the chance of a major journal publication. By insisting on pre-registration, they were doing what they could to force researchers and industry sponsors to register all trials. Everyone rejoiced: the problem had been fixed.
If you think it seems odd – and perhaps unrealistic – that fixing this crucial flaw in the information architecture of a $700 billion industry should be left to an informal gathering of a few academic editors, with no legislative power, then you’d be right. Although everybody began to talk as if publication bias was a thing of the past, in reality it was continuing just as before, because the journal editors simply ignored their own threats and promises. Later (pp.247, 307) we will see the phenomenal financial inducements on offer to editors for publishing positive industry papers, which can extend to millions of dollars in reprint and advertising revenue. But first we should look at what they actually did after their solemn promise in 2005.
In 2008 a group of researchers went through every single trial published in the top ten medical journals, every one of which was a member of the ICMJE, after the deadline for pre-registration. Out of 323 trials published during 2008 in these high-impact academic journals, only half were adequately registered (before the trial, with the main outcome measure properly specified), and trial registration was entirely lacking for over a quarter.63 The ICMJE editors had simply failed to keep their word.
Meanwhile, in Europe, there were some very bizarre developments. With enormous fanfare, the European Medicines Agency created a registry of trials called EudraCT. EU legislation requires all trials to be posted here if they involve any patients in Europe, and many companies will tell you that they’ve met their responsibilities for transparency by doing so. But the contents of this EU register have been kept entirely secret. I can tell you that it contains around 30,000 trials, since that figure is in the public domain, but that is literally all I know, and all anyone can know. Despite EU legislation requiring that the public should be given access to the contents of this register, it remains closed. This creates an almost laughable paradox: the EU clinical trials register is a transparency tool, held entirely in secret. Since March 2011, after heavy media criticism (from me at any rate), a subset of trials has slowly been made public through a website called EudraPharm. As of summer 2012, although the agency now claims that its register is accessible to all, at least 10,000 trials are still missing, and the search engine doesn’t work properly.64 It’s absolutely one of the strangest things I’ve ever seen, and nobody other than the EU even regards this peculiar exercise as a trials register: I certainly don’t, I doubt you do, and both the ICMJE and the World Health Organization have explicitly stated that EudraCT is not a meaningful register.
But new work was being done in the US, and it seemed sensible. In 2007 the FDA Amendment Act was passed. This is much tighter: it requires registration of all trials of any drug or device, at any stage of development other than ‘first-in-man’ tests, if they have any site in the US, or involve any kind of application to bring a new drug onto the market. It also imposes a startling new requirement: all results of all trials must be posted to clinicaltrials.gov, in abbreviated summary tables, within one year of completion, for any trial on any marketed drug that completes after 2007.
Once again, to great fanfare, everyone believed that the problem had been fixed. But it hasn’t been, for two very important reasons.
Firstly, unfortunately, despite the undoubted goodwill, requiring the publication of all trials starting from ‘now’ does absolutely nothing for medicine today. There is no imaginable clinic, anywhere in the world, at which medicine is practised only on the basis of trials that completed within the past three years, using only drugs that came to market since 2008. In fact, quite the opposite is true: the vast majority of drugs currently in use came to market over the past ten, twenty or thirty years, and one of the great challenges for the pharmaceutical industry today is to create drugs that are anything like as innovative as those that were introduced in what has come to be known as the ‘golden era’ of pharmaceutical research, when all the most widely used drugs, for all the most common diseases, were developed. Perhaps they were the ‘low-lying fruit’, plucked from the research tree, but in any case, these are the tablets we use.
And crucially, it is for these drugs – the ones we actually use – that we need the evidence: from trials completed in 2005 or 1995. These are the drugs that we are prescribing completely blind, misled by a biased sample of trials, selectively published, with the unflattering data buried in secure underground data archives somewhere in the hills (I am told) of Cheshire.
But there is a second, more disturbing reason why these regulations should be taken with a pinch of salt: they have been widely ignored. A study published in January 2012 looked at the first slice of trials subject to mandatory reporting, and found that only one in five had met its obligation to post results.65 Perhaps this is not surprising: the fine for non-compliance is $10,000 a day, which sounds spectacular, until you realise that it’s only $3.5 million a year, which is chickenfeed for a drug bringing in $4 billion a year. And what’s more, no such fine has ever been levied, throughout the entire history of the legislation.
So that, in total, is why I regard the ICMJE, the FDA and the EU’s claims to have addressed this problem as ‘fake fixes’. In fact, they have done worse than fail: they have given false reassurance that the problem has been fixed, false reassurance that it has gone away, and they have led us to take our eyes off the ball. For half a decade now, people in medicine and academia have talked about publication bias as if it was yesterday’s problem, discovered in the 1990s and early 2000s, and swiftly fixed.
But the problem of missing data has not gone away, and soon we will see exactly how shameless some companies and regulators can be, in the very present day.
What can be done?
The ICMJE should keep its promises, the EU should be less risible, and the FDA Amendment Act 2007 should be muscularly enforced. But there are many more disappointments to come, so I will leave my action plan on missing data for later in this chapter.
Blood from a stone: trying to get data from regulators
So far we’ve established that doctors and patients have been failed by a range of different people and organisations, all of whom we might have expected to step up and fix the problem of missing data, since it harms patients in large numbers around the world. We have seen that governments take no