2. Assessing Bias in Randomized and Non-Randomized Studies: New Approaches, New Tools

Detailed course information (PDF)


Prof. Jonathan Sterne (course co-ordinator)
School of Social and Community Medicine, University of Bristol, United Kingdom

Prof. Julian Higgins
School of Social and Community Medicine, University of Bristol, United Kingdom


Randomized controlled trials (RCTs), and systematic reviews of such trials, provide the most reliable evidence about the effects of healthcare interventions. Providing enough participants are randomized, randomization should ensure similarity of participants in the intervention and comparison groups so that differences in outcomes of interest between these groups can be ascribed to the causal effect of the intervention. Causal inferences from RCTs can, however, be undermined by flaws in design, conduct, analyses and selective reporting. Although there is good empirical evidence that flaws in RCTs may lead to bias, it is usually impossible to know the extent to which biases have affected the results of a particular trial. Therefore systematic reviews of RCTs typically include assessments of the validity of the included trials.

Non-randomized studies of interventions (NRSI) can provide evidence additional to that available from RCTs about long-term outcomes, rare events, adverse effects and populations that are typical of real world practice. For many types of organizational or public health interventions, NRSI are the main source of evidence about the likely impact of the intervention because RCTs are difficult or impossible to conduct on an area-wide basis. Therefore systematic reviews addressing the effects of healthcare interventions often include NRSI.

In the last decade, major developments have been made in tools to assess study validity. A shift in focus from methodological quality to risk of bias has been accompanied by a move from checklists and numeric scores towards domain-based assessments in which different types of bias are considered in turn. Examples are the Cochrane Risk of Bias tool for randomized trials, the QUADAS 2 tool for diagnostic test accuracy studies and the ROBIS tool for systematic reviews.

This course introduces two newly-developed tools for assessing the risk of bias: version 2 of the Cochrane tool for assessing risk of bias in RCTs, and the ROBINS-I tool for assessing risk of bias in NRSI. These tools share similar approaches including the use of signalling questions to help reviewers judge the risk of bias within each domain, specification of the effect of interest, and guidance on assessing the overall risk of bias in a particular study result. However some of the bias domains assessed differ between the tools: for NRSI but not RCTs it is necessary to assess the risk of bias due to confounding; selection bias; and bias in classification of interventions. Work extending ROBINS-I to assess studies of exposures will be described.

Course objectives

By the end of this short course participants will:

  • Understand the empirical and theoretical evidence for bias in RCTs and NRSI
  • Understand the types of bias that can undermine the internal validity of RCTs and NRSI
  • Be able to use version 2 of the Cochrane tool to assess risk of bias in RCTs
  • Be able to use the ROBINS-I tool to assess risk of bias in NRSI

Course fees

Academic fee: CHF 900
Industry fee: CHF 2’000
SSPH+ fee: only applicable for students of the SSPH+ PhD Program in Public Health