Overview
Ensuring that systematic reviews and meta-analyses include a representative and unbiased collection of relevant studies is fundamental to producing reliable conclusions. Traditionally, extensive searches covering grey literature, multiple databases, and non-English publications are recommended to minimise reporting bias. However, such approaches require considerable time and financial resources, and their actual influence on review outcomes remains uncertain. Balancing thoroughness with practicality is therefore an important methodological consideration.
Study Aims
This investigation sought to:
- Characterise clinical trials that are harder to identify, including unpublished studies, non-English publications, and trials appearing in journals not indexed in MEDLINE.
- Examine features of lower-quality trials, particularly those lacking proper allocation concealment or double-blinding.
- Compare treatment effect estimates within meta-analyses between:
- Hard-to-find vs. easily accessible trials
- Lower-quality vs. higher-quality trials
- Determine how excluding such trials affects pooled estimates, statistical significance, and funnel plot symmetry.
Methods
Sources of Data
Relevant meta-analyses were identified through:
- Manual searches of eight medical journals known for publishing systematic reviews
- The Cochrane Database of Systematic Reviews
- The Database of Abstracts of Reviews of Effectiveness
- Health Technology Assessment publications (handsearched)
Inclusion Criteria
Eligible meta-analyses:
- Evaluated therapeutic or preventive interventions
- Included at least five controlled trials with binary outcomes
- Conducted comprehensive searches, defined as:
- No restriction to English-language publications
- Searching at least two electronic databases (e.g., MEDLINE, EMBASE) or the Cochrane Controlled Trials Register
- Evidence of attempts to locate unpublished studies
Data Handling
Trials were categorised by publication status, language, indexing status, and methodological quality. Quality assessments were limited to trials within Cochrane reviews.
For each eligible meta-analysis, pooled treatment effects were calculated separately for accessible vs. difficult-to-locate trials using the original statistical model. Ratios of pooled estimates were computed and combined using random-effects meta-analysis. The impact of excluding certain trials on effect size, p-values, and funnel plot asymmetry was also evaluated.
Findings
A total of 159 systematic reviews met the inclusion criteria. Comparisons were based on:
- Published vs. unpublished trials (60 meta-analyses)
- English vs. non-English trials (50 meta-analyses)
- MEDLINE-indexed vs. non-indexed trials (66 meta-analyses)
Quality-related analyses included:
- Adequate vs. inadequate/unclear allocation concealment (39 meta-analyses)
- Double-blind vs. non-double-blind trials (45 meta-analyses)
Key Observations
- The relevance of hard-to-locate trials varies by medical discipline.
- Unpublished trials were especially common in oncology.
- Non-English and non-indexed trials were more influential in psychiatry, rheumatology, and orthopaedics.
- Complementary medicine frequently relied on studies that were difficult to locate.
In general:
- Unpublished trials reported less favourable treatment effects than published trials.
- Non-English and non-indexed studies often showed larger effects.
- Hard-to-find studies tended to be smaller and methodologically weaker.
- Trials with inadequate allocation concealment or lacking blinding frequently overstated benefits.
Excluding lower-quality trials usually reduced the estimated treatment effect, often substantially.
Including unpublished trials reduced funnel plot asymmetry, while adding non-English or non-indexed studies increased asymmetry. Lower methodological quality, particularly unclear allocation concealment, had a strong effect on small-study bias indicators.
Conclusions
Reviews limited to English-language studies indexed in major databases often yield results comparable to those derived from more exhaustive searches. When designing a systematic review, researchers should weigh the need for comprehensiveness against available time and resources.
Importantly, including methodologically weak studies in the name of completeness may introduce bias rather than eliminate it. When resources are constrained, careful assessment of trial quality may be more valuable than extensive searching and translation efforts.
The study also confirms that funnel plots and regression-based asymmetry tests are useful tools for detecting small-study effects.