Bias in Publication and Reporting
Bias in Publication and Reporting
Abstract and Keywords
This chapter discusses situations where the hypothesis testing construct is deliberately violated by not publishing studies that fail to support a preferred hypothesis, by reporting them in a manner which conceals what the actual hypothesis was, or, even worse, by misrepresenting the original hypothesis. Hypotheses are not discarded but rather are transmuted during publication to something entirely different, and in so doing, public safety is severely jeopardized. The chapter defines the concept of “file drawer bias,” a tendency of academic as well as corporate investigators to write up and submit for publication results that favor their employer's product, their long-held hypotheses, or their perceived chances of achieving tenure. This is one aspect of a much larger set of problems in reporting studies: “publication bias.”
The great tragedy of science is the slaying of a beautiful hypothesis by an ugly fact.
—Thomas Henry Huxley, 1870
Don't confuse hypothesis and theory. The former is a possible explanation; the latter, the correct one. The establishment of theory is the very purpose of science.
—Martin H. Fischer
Traditionally, scientific knowledge has been obtained by developing hypotheses that can be tested in a study or experiment designed to produce data that will either support or refute it. As Huxley satirically put it in his presidential address to the British Association for the Advancement of Science, science is littered with beautiful but failed hypotheses. In the modern medical sciences, we are less reliant on the results of a single study, preferring to see replication in several of them and their confirmation within the framework of a systematic review and meta-analysis. Nonetheless, the scientific paradigm of hypothesis testing prevails: the hypothesis is typically laid out in a study protocol to describe exactly what is being tested and the precise outcome expected. It is not easy for scientists to discard what often becomes a favored and long-held hypothesis. Konrad Lorenz, the influential behavioral zoologist, advised: “It is a good morning exercise for a research scientist to discard a pet hypothesis every day before breakfast. It keeps him young.”
(p.225) In this chapter, we discuss situations where the hypothesis testing construct is deliberately violated by not publishing studies that fail to support a preferred hypothesis, by reporting them in a manner that conceals what the actual hypothesis was, or, even worse, by misrepresenting the original hypothesis. Hypotheses are not discarded but rather are transmuted during publication to something entirely different, and in so doing, the safety of the public is severely jeopardized.
Some years ago I was asked by a very large pharmaceutical company to review the evidence supporting a glossy advertising campaign being run in the leading medical journals touting the claim by another equally large company that its antihypertensive drug was superior to that of my client's. Reference was made in the advertisement to a small, randomized trial of only 32 subjects, which documented a marginal clinical improvement in the other company's product compared with my client's drug. The trial was unimpressive, clearly subject to being wrong, and had any of the prescribing doctors to whom the expensive advertising was aimed bothered to look up the citation, they assuredly would not have been impressed. During the pretrial depositions, it was discovered that a second (equally unimpressive) trial of similar size had been conducted and was found residing in a file drawer with no plans to publish. Perhaps not surprisingly, this trial showed an equally modest improvement, but this time the benefit was to the patients treated with my client's drug. A simple meta-analysis combining data from the two trials showed absolutely no difference in the clinical effectiveness of the two antihypertensive drugs. The other company's drug was not superior to that of my client's. The Food and Drug Administration (FDA), which has regulatory control over pharmaceutical company advertising, was extremely annoyed, and the lawsuit between the companies was quickly settled.
What had been observed in the above anecdote is a classic (and literal) example of what is called “file drawer bias”; this is the inclination of investigators (academic as well as corporate) to write up and submit for publication those results that favor their employer's product, their long-held hypotheses, or their perceived chances of achieving tenure. Not only would the doctors examining the citation to the antihypertensive medication not have been (p.226) impressed with the paper that was cited, they would have had no way of knowing that another trial, producing the opposite result, had been conducted. File drawer bias is one aspect of a larger set of problems in reporting studies: “publication bias.” This is a particularly pernicious source of bias when investigators are conducting a meta-analysis that seeks to summarize the totality of evidence on any particular topic. If the literature is biased by selective publication of positive results, then conclusions drawn about the effectiveness of therapies will be wrong. This bias can have life-threatening consequences.
Hexamethonium is a blood pressure medication that was used in a study funded by Johns Hopkins University and the National Institutes of Health to try and understand how asthma symptoms developed. The study had to be suspended when one of its participants, a previously healthy 24-year-old woman, died of lung collapse soon after taking the medication. The investigation committee, established by Johns Hopkins to look into the death, examined how thorough a literature search was conducted by the investigator to identify potential drug complications before he submitted his research protocol to the ethics review committee. It turned out, using a standard PubMed search and consulting current textbooks, that some key studies had been missed. The ethics committee, after the research volunteer's death, conducted a search of several bibliographic databases and in reviewing material from the 1950s found earlier work suggesting pulmonary side effects from hexamethonium. In a letter to the NIH oversight office, a university dean wrote, “The committee was divided on the issue of what constitutes a sufficient search of the literature in support of a human subjects research application.”1 Even in 2011, a search using the Johns Hopkins investigator's name and hexamethonium produced no results in PubMed, but it found eleven using Google Scholar, a reflection of the inadequacy of relying solely on a PubMed search.2 One can only speculate that perhaps it is due to legal advice that no complete report of this failed experiment can be found in the literature. Hopefully, no one in the future will attempt to replicate this failed experiment, but if they do, inadequate reporting of the first study can only increase the chance of repeated disaster.
(p.227) The proclivity for pharmaceutical companies to publish only the results of studies that favor their products was formally investigated by a group of scientists interested in antidepressant drugs. Using an innovative strategy, they obtained the results of trials submitted by pharmaceutical companies for review to the FDA to seek licensing approval. Some of the information was obtained under the US Freedom of Information Act and is unlikely to have otherwise seen the light of day. The results of the trials in the FDA files were then compared to published reports in the medical literature. In all, 74 FDA-registered studies were located concerning 12 antidepressant agents and 12,564 patients. Whether or not the drug was found to be efficacious in treating depression, and the size of the benefit, was judged using the FDA's own determination during the regulatory process.
The results were striking: 38 trials met the criterion for a positive effect and 37 were published. In contrast, of the 36 trials considered negative or questionable by the FDA, only 3 were subsequently published. In short, the positive studies were 12 times more likely to be published than the negative ones. The net effect of this was that the published literature suggested 94 percent of the trials were positive but the FDA analysis indicated only 51 percent as being positive. Even among the published trials found to be positive by the FDA, the published effect size was on average 32 percent greater in the published reports than obtained in the FDA review. Were the unpublished studies inferior in some respect that might explain their lack of publication? The review authors could find no evidence for this: the studies were of similar size, and all the protocols had been written to meet international guidelines for drug efficacy studies. Finally, it was noted that this, quite depressing, bias in publication occurred among all the drugs studied and all the studied pharmaceutical companies.3
It has been estimated that some 25 to 50 percent of all studies are never published; in one project Kay Dickersin, one of the leading investigators into publication bias, followed up on studies whose protocols were submitted to ethics committees for review at several institutions and found that 8 to 14 years later 7 to 45 percent remained unpublished. The fewest unpublished trials were among the large NIH-sponsored clinical trials. When (p.228) investigators were asked why their results were unpublished, the most frequently cited reason was that the results were “uninteresting.”4 In another study asking scientists why they delayed publication of their results for more than six months, the most frequent replies were to allow time for patent application, because of the proprietary or financial value of results, or to preserve the investigators' scientific lead, but 28 percent admitted it was intended to delay dissemination of undesired results.5
It is known that those studies that are published tend to favor what are called “positive” results; that is, results that find a definite answer to the research question, usually an increased risk for a suspected environmental exposure or a benefit for therapeutic research. Publication bias, therefore, is defined as the tendency toward the preparation, submission, and publication of research findings, based on the nature and direction of the results. As the definition indicates, it occurs across the publication spectrum: from the investigators' eagerness or not to write up a study report, deciding to which journal to submit it, and to the editor's willingness to accept the paper for publication. It has been found that statistically significant published study results are reported more frequently in journals with higher-impact factors (that is, their articles are cited by more researchers). Similarly, more statistically significant results are also found in higher-impact journals.6
John Ioannidis reported that all but a few “positive” trials were submitted for publication as late as two and a half years after completion but as many as one-fifth of negative trials remained unsubmitted even five years after completion. After submission, the median time for a positive trial to publication was just under one year but for a negative trial it was a few months longer.7 Stern and Simes examined publication delay in a variety of quantitative studies from the time they were submitted for ethics committee approval to publication. Studies reporting statistically significant results fared best: only 1.4 percent remained unpublished after 10 years compared with 13.5 percent of the statistically nonsignificant studies.8
It is also known that only a small proportion of citations recovered by electronic searching of the major literature databases are relevant for the inquiry at hand. Clinical trials are particularly difficult to find; only about half of all relevant trials are successfully retrieved in a typical search on (p.229) the largest database, MEDLINE.9 This is largely due to incomplete indexing of RCTs so that searching for text words (which is usually restricted to the abstract) can fail to detect a randomized study. This limitation has forced the Cochrane Collaboration, described earlier as the leading organization that maintains a register of reports of clinical trials, to hand search hundreds of journals so that more reports of clinical trials can be correctly identified.
Another aspect of publication bias concerns the language in which a paper has been written. Research has shown that non-English-language papers are of similar quality to those written in English; however, they are more likely to have statistically nonsignificant results. This is exemplified by a hypothetical trial result at Stockholm University—if the results are positive and considered to be important the paper is likely sent to one of the four most widely read general clinical journals (the Lancet, BMJ [formerly British Medical Journal], New England Journal of Medicine, and JAMA [Journal of the American Medical Association]), all of which publish in English. If the results are considered null and perhaps not very interesting, they may be more likely to be submitted to a Swedish journal publishing in that language. Not only are non-English-speaking journals less accessible on MEDLINE and other large bibliographic research databases, searches on MEDLINE are often restricted to the English-language journals. Imagine how many important studies of acupuncture would be missed if those in non-English languages were to be excluded from a literature search.
Other pertinent research reports may be found in what is called the gray literature. These are internal company or foundation reports, industry files, and academic dissertations. Some research reports are known only because of personal knowledge. Abstracts of presentations at scientific meetings are another source of information although care in interpreting them is warranted: many investigators have to produce an abstract to be able to attend conferences, but the abstract may represent incomplete, premature, or biased results that will not be confirmed when the study is finally completed.
We started this chapter with examples whereby unfavorable trial results were being withheld, and an incomplete literature search had fatal consequences for a research volunteer. Publication bias is important for more (p.230) (p.231) prosaic reasons. First, we rely on the published literature to represent the true nature of current knowledge. Scientific progress is substantially hindered if the complete picture of what is currently known is perverted; this will lead to unnecessary replication and possible delay in making new therapeutic advances. Science, even at its best, goes through many twists and turns, false leads, and dead ends; it does not need the added complication of a misleading representation of the extant literature. Second, systematic reviews of the published literature are the source for clinical guidelines and policy statements. If the literature being used is a biased and incomplete picture of the full state of knowledge, the guidelines and policy may be wrong, resulting in needless risk and hazard for individual patients and the public.
One example of this concerns pregnant women with Group B streptococcal infections. Several years ago, a Centers for Disease Control guideline committee promulgated a recommendation that all women about to deliver a baby be tested for Group B strep and those testing positive (about 20 percent of the population) be given antibiotics for at least four hours before delivery. If a four-hour course could not be given, typically because the woman went into labor and delivered, it was recommended that the newborn have blood cultures drawn and be placed under observation, often in the newborn intensive care unit (NICU) or other observation area. This guideline resulted in tens of thousands of babies being separated from their mothers immediately after birth and placed in a NICU.
My colleague Jessica Illuzzi, a young academic obstetrician, took the bold step of questioning this guideline by reexamining the guideline committee's work. She found no evidence that it had conducted a systematic review of the relevant literature to inform its deliberations, and so Jessica did one herself: it provided no evidence to support the four-hour course of antibiotics.10 In other work, she found that antibiotic levels could be achieved after just several minutes of administration. It was several years before the guideline committee reconvened, but based on a more extensive literature search, its recommendation for therapy was considerably modified to simply advising a four-hour period of therapy. This is likely to result in far fewer newborns undergoing invasive testing and being unnecessarily placed in an intensive care unit.
We now turn to a type of publication bias in which a study is published but the original study hypothesis is misrepresented in the published report. The aspect of the study that is most susceptible to being changed is in reporting the study outcomes. This has been found to occur when reporting the results of a clinical trial where the prespecified primary (the most important) study outcome is ignored and another outcome is reported as if it had been the preplanned primary outcome. Surprisingly, this phenomenon (technically called “outcome reporting bias”) was only first formally studied in 2004 by An-Wen Chan who compared the published reports of randomized controlled trials and what was stated to be the trial's primary and secondary clinical outcomes with the previously declared outcomes in the trial protocol.11
When a trial protocol is prepared, the investigators list one or two of the most important clinical outcomes that they hypothesize will be influenced by the therapy under study, based on all the experimental work that has preceded their trial. They also list a longer series of secondary outcomes that are of interest but not considered to be the most important outcomes. These are more in the “if we get lucky” category because compared to the primary outcomes, they are not as well founded in the earlier science. If a therapy is found to influence only a secondary outcome, it is not usually considered to be a definitive demonstration of the therapy's effectiveness.
Astonishingly, Chan found that almost two-thirds of the published reports were discrepant with the stated goals of the trial as written in the protocol: in 34 percent the primary outcome in the protocol was published as a secondary outcome; for 26 percent the protocol's primary outcome was not even reported in the publication; 19 percent of studies changed a secondary outcome in the protocol to make it primary in the report; and 17 percent published as a primary outcome something not even mentioned in the protocol.
Among the examples documented by Chan were trials where the proportion of patients with severe cerebral bleeding was changed from a primary to a secondary outcome, the primary outcome of event-free survival was omitted (p.233) from the published report, an overall symptom score was changed from a secondary outcome in the protocol to primary in the publication, and the proportion of patients with a graft occlusion was not mentioned in the protocol but listed as the primary outcome in the report. Overall, 50 percent of the efficacy outcomes listed in the protocol and 65 percent of the harm outcomes were incompletely reported in the publication. Why is this important? Having read this far into the book, readers will not be surprised to learn that the statistically significant outcomes were being reported in a disproportionate manner; in fact, statistically significant outcomes pertaining to the effectiveness of the treatment were almost two and a half times more likely to be reported than nonsignificant ones. For the outcomes concerning possible harm from the therapy, the bias was even higher: statistically significant outcomes were almost five times more likely to be reported than nonsignificant ones.
It is axiomatic in science that a specific intervention is predicted to cause a precise outcome or event. This is described in the hypothesis being tested and there is no room for variability. The hypothesis must be tested, and then based on the data it will be rejected or not rejected (it cannot be formally proven, see chapter 6, but we can informally say it has been “accepted”). If the hypothesis is accepted, it becomes a building block toward a more general theory of the phenomenon under study. If the hypothesis is rejected, new hypotheses are proposed for future testing. If the primary hypothesized outcome was not statistically significant but another outcome was, then the hypothesis has been rejected and a new hypothesis concerning the successful outcome can be proposed. The difference is not subtle: a new hypothesis cannot be accepted in a study of a prior hypothesis, the new hypothesis can only be generated. It must be tested in a new study. What Chan uncovered was the deliberate perversion of the scientific process by relabeling what may well have been a spurious but statistically significant finding as if it were testing a primary hypothesis.12
Stephen Jay Gould, the Harvard evolutionary biologist, pointed to the invidious effect of publication bias on the scientific process: “In publication bias, prejudices arising from hope, cultural expectation, or the definitions of a particular theory dictate that only certain kinds of data will be (p.234) viewed as worthy of publication, or even of documentation at all. Publication bias bears no relationship whatever with the simply immoral practice of fraud; but paradoxically, publication bias may exert a far more serious effect (largely because the phenomenon must be so much more common than fraud)—for scientists affected by publication bias do not recognize their errors (and their bias may be widely shared among colleagues), while a perpetrator of fraud operates with conscious intent, and the wrath of a colleague will be tremendous upon any discovery.”13 Disappointingly, Chan found that 86 percent of authors denied any existence of unreported outcomes despite clear evidence to the contrary in the protocol.
In the study of antidepressants, described earlier in this chapter, it was also noted that the methods sections in 11 of the 37 published papers differed from the prespecified methods described for the FDA. In every study, a positive result was reported as being the primary outcome. When the prespecified primary outcome was reported as being negative, in two reports it was relegated to having a secondary status or it was completely omitted (in nine reports).14
Duplicitous and Duplicate Publication
When scientists submit a paper for publication they must assure the journal's editor that the paper has not been previously published and that it is not being considered for publication by any other journal. The intent is to avoid essentially identical reports from entering the research literature. The reasons for this are twofold. Elsewhere we have discussed the importance of replication as a tool for validating research findings; this implies that the replicate research projects are independent of each other. Ideally they involve different investigators, but certainly they must include unique study subjects. There are differing degrees of duplication, ranging from the egregious examples of the same paper being published in more than one journal—usually reflecting an effort (that almost always fails) to pad an academic curriculum vitae, to the appearance of the same study subjects, sometimes innocently, in multiple study reports. The former is more properly (p.235) called “duplieitous publication,” and we will discuss several examples of duplicity and duplication below.
One common duplicitous strategy is to publish a paper in another language and then to republish in English, or vice versa, sometimes with a rearrangement of the author list. When I confronted the French author of a paper that had been republished in English, I was told it was for two reasons, both specious—to give another author a chance at first authorship and to make the paper more widely available. An author deserves to be listed first on a paper only if he or she earned it, by leading the research; the language is irrelevant. Moreover, there are ways of highlighting the work of two authors that is considered of equal importance (these were not used in this paper). Further, the availability of electronic databases makes almost all the world's research literature available. It is revealing that in this example the earlier French publication was not cited in the later English one, making it questionable whether the English editor was informed of the existence of the earlier publication.15
How common is duplicate publication? This has been examined in a website creatively named Déjà Vu that collects examples of publication duplication although the website editors seem more interested in the duplicitous variety. In one analysis of a sample of 62,213 citations in MEDLINE using software for comparing text, 1.35 percent of papers with the same authors were judged to be duplicates and 0.04 percent had different authors, raising the specter of possible plagiarism.16 Less duplicitous examples of publication duplication are more difficult to identify, but they pose problems for scholars trying to synthesize research literature or conducting meta-analyses.
Pregnant women who are epileptic need to be medicated to control their epilepsy symptoms, but the obvious question arises: does this incur complication to the fetus from the therapy? Since only about 1 percent of pregnant women are epileptic and congenital malformations are rare, the usual research practice is to examine the records of birth defect registries. In 1999 a Dr. F. J. E. Vajda and colleagues founded the Australian Register of Antiepileptic Drugs in Pregnancy to address this question, and since then at (p.236) least half a dozen papers have been published from the accumulating data in the registry.17 There is nothing improper about the regular publication of data from registries of this type provided prior publications are cited and the authors are always explicit about their earlier work, as the Australian investigators are. Indeed, it is laudable that they are trying to keep the published information from the registry as current as possible. However, researchers who are synthesizing research data need to be aware that the birth defects cases reported in the first publications continue to be reported in the later ones; the reports are not independent.
Similar issues arise when the large Scandinavian databases, which are excellent resources for examining the association of rare exposures with uncommon discases, are used. In 2007, the well-respected epidemiologist Dr. Bengt Källén published a study linking three Swedish registries (births, birth defects, and patients) to examine the question of whether antidepressant drugs when used by pregnant women increased their risk of having a congenitally malformed child. In this publication over 860,000 children were studied. A 2010 study of over 1 million children by Dr. Margareta Reis, a colleague of Dr. Källén and using the same databases, included all the children from the earlier paper. Again, the authors were careful to point this out but careless readers may assume these were two very large and independent studies.18 Certainly, they have been used in this way, knowingly or not, by unscrupulous expert witnesses for plaintiffs in lawsuits on this matter (disclosure, I have been a consultant on the opposite side). Meta-analysts of these types of study must be aware to include in their analyses only the most recent data from a sequence of reports from the same database.
More difficult to delineate are what appear to be independent reports that may include the same patients. This occurs particularly when rare outcomes are being studied. An interesting group of uncommon cases and their exposures may be written up by hospital staff, the same cases may be reported to a case registry where a different group of doctors include them in a publication of a larger series of cases on the same exposure, and they may also form part of a national data set where the cases are again published—all without reference to the earlier publications. This may have happened in 2007 for two seemingly independent and highly publicized (p.237) reports published together in the New England Journal of Medicine which also examined the risk of congenital malformation in mothers using antidepressants. One study included cases from hospitals in the cities of Boston, New York, and San Diego and the other study used cases from state birth defect registries emanating from Massachusetts, California, and New York, with the time period for births covered in one study fully overlapping the births in the other. It seems highly likely that the cases from the hospitals in one study were also listed in the state registries.19
Interest in these types of study, if it is to inform patient management and address issues of causality, has to ultimately focus on specific drugs and examine particular malformations. The evidence for an association invariably rests on a very small number of exposed cases, and the actual number can affect how one views causality. In this example, interest has focused on the drug paroxetine (Paxil) and right ventricular outflow tract obstructive (RVOTO) defects that occur in the heart. The statewide registries reported six babies with RVOTO who had been exposed to paroxetine and the hospital-based study one. Whether this last case was included in the six is unknown but seems likely. If the same cases are being reported in different studies, then there is no longer replication of the observation and an important criterion for causality is lost. This point was not mentioned by the journal's editorial writer in an otherwise perceptive commentary and was also missed, one suspects, by the journal editors themselves. It was certainly not picked up by the journalists covering the story.20
The duplication problem arising from the same cases being reported many times in multiple data sets will only increase. There is now a strong mandate for investigators to make their data sets broadly available so that they can be reanalyzed and republished, and genetic data sets are already being publicly archived, repeatedly reanalyzed, and published many times, making it difficult to track their publication history. Scientists will have to pay more attention to the problem of the same data reoccurring in multiple papers when they try to weight the overall evidence for a particular risk association.
In this chapter we have seen that publication bias is a quite frequent phenomenon. Not only does it pervert the scientific process, contributing (p.238) to the adoption of ineffective and possibly harmful therapies, but it can have fateful consequences. Fortunately, in recent years, recognition of the insidious influence of publication bias has resulted in the creation of electronic databases for the registration of clinical trial protocols. These allow the progress and eventual publication of trial results to be tracked and comparisons made with those study outcomes which were initially proposed in the protocol and what is reported at the end of the study.21 In observational studies that are examining potential harms, there has been less progress and some opposition to the registration of study protocols although the imperative to do so is no less important.22
(*) Adapted by permission from BMJ Publishing Group Limited. (How citation distortions create unfounded authority: analysis of a citation network, Greenberg SA, 339:b2680, Copyright 2009.)
(1) . Hopkins issues internal review committee report on research volunteer's death. Johns Hopkins Medicine website. http://www.hopkinsmedicine.org/press/2001/JULY/010716.htm. July 16, 2001. Accessed April 26, 2012.
(2) . Other searches in PubMed do produce a large literature on hexamethonium. Searched May 29, 2011. MEDLINE is the database published by the National Library of Medicine; PubMed is the part of it typically used to search the literature because it includes journal articles before they have been formally published in a print journal. MEDLINE has highly sophisticated technical strategies for indexing articles to help searching. As of May 2011, it included over 20 million articles in the biomedical and life sciences literature and electronic books.
(3) . Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252–260.
(4) . Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA. 1992;267(3):374–378.
(5) . Blumenthal D, Campbell EG, Anderson MS, Causino N, Louis KS. Withholding research results in academic life science: evidence from a national survey of faculty. JAMA. 1997;277(15):1224–1228. 410 per 2,167 (19.8 percent) US life science faculty responding to a 1995 survey reported delaying publication beyond six months.
(6) . Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337(8746):867–872; Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007;(2):MR000011.
(7) . Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998;279(4): 281–286.
(8) . Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315(7109):640–645.
(9) . Hopewell S, Clarke M, Lefebvre C, Scherer R. Hand searching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev 2007;(2):MR000001.
(10) . Illuzzi JL, Bracken MB. Duration of intrapartum prophylaxis for neonatal Group B streptococcal disease: a systematic review. Obstet Gynecol. 2006; 108(5):1254–1265.
(11) . Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–2465.
(13) . Gould SJ. The structure of evolutionary theory. Cambridge: Harvard University Press; 2002:763–764.
(15) . Petitjean ME, Pointillart V, Dixmerias F, et al. Traitement medicamenteux de la lesion medullaire traumatique au stade aigu. Ann Fr Anesth Reanim. 1998;17(2):114–122; Pointillart V, Petitjean ME, Wiart L, et al. Pharmacological therapy of spinal cord injury during the acute phase. Spinal Cord. 2000;38(2):71–76.
(16) . Errami M, Hicks JM, Fisher W, et al. Déjà vu--a study of duplicate citations in Medline. Bioinformatics. 2008;24(2):243–249.
(17) . The most recent paper in the series at time of writing is Vajda FJ, Graham J, Hitchcock AA, O'Brien TJ, Lander CM, Eadie MJ. Foetal malformations after exposure to antiepileptic drugs in utero assessed at birth and 12 months later: observations from the Australian pregnancy register. Acta Neurol Scand. 2011;124(1):9–12.
(18) . The most recent paper is Reis M, Källén B. Delivery outcome after maternal use of antidepressant drugs in pregnancy: an update using Swedish data. Psychol Med. 2010;40(10):1723–1733.
(19) . Louik C, Lin AE, Werler MM, Hernández-Díaz S, Mitchell AA. First-trimester use of selective serotonin-reuptake inhibitors and the risk of birth defects. N Engl J Med. 2007;356(26):2675–2683, and Alwan S, Reefhuis J, Rasmussen SA, Olney RS, Friedman JM; National Birth Defects Prevention Study. Use of selective serotoninreuptake inhibitors in pregnancy and the risk of birth defects. N Engl J Med. 2007;356(26):2684–2692.
(22) . Bracken MB. Preregistration of epidemiology protocols: a commentary in support. Epidemiology. 2011;22(2):135–137.