If no patients have complications in a small study, how confident can we be that a medical procedure is safe?

Details

This statistical and ethical commentary discusses a study by Fabregat-López and colleagues that also raises questions about airway management. They used a Proseal laryngeal mask airway (PLMA) during laparoscopic emergency appendicectomy, apparently a routine practice in their unit for a considerable time. From a UK perspective, the approach could appear a gamble, but they reported no complications. To investigate their belief that PLMA use was safe in these circumstances, they conducted a single-group observational study: one group of patients broadly received the same technique. Their reasoning—that few or no complications would establish safety—prompts the commentary's central questions: is that inference justified, and how many patients must be observed to be reasonably confident about safety?

The commentary distinguishes this question from the power calculations discussed in previous anaesthesia articles and editorials for comparisons between two techniques. Those same statistics cannot simply be applied to complication incidence in one patient group. Instead, it discusses Poisson and binomial distributions and probability density functions, describing these as closely related. An appendix sets out the mathematics; the main discussion aims to explain its purpose in simple, practical terms.

Its illustrative example is watching birds pass a window. After many weeks, an average of 12 birds per 24 h would imply averages of 24 birds over two days or six over half a day. Those averages alone do not answer the probability of seeing at least one bird in 1 h, or how long someone must watch to be confident of seeing 36 birds. The commentary applies the same mathematics to complications or beneficial outcomes in populations. If a complication occurs on average in 1 in every 200 cases, how many cases must be observed to be reasonably confident of seeing at least one? Conversely, among 150 observed cases, what is the probability of seeing at least one complication, or none? These are illustrative questions, not reported patient outcomes.

The appendix's mathematics turns these event frequencies into a probability distribution or density. With no observations, the probability of observing any events is zero; with infinitely many observations, the commentary describes that probability as 1. Between those extremes, the probability depends on observation time or number of observations and on how common the event is. The discussion also specifies conditions: events must be independent, so one event does not influence the chance of another, and must be discrete yes/no observations, such as a complication being present or absent. Continuous measurements such as blood pressure or height are not suitable for this analysis. Computer algorithms can make the appendix's formula easier to use; the examples supplied are http://www.stat.tamu.edu/~west/applets/binomialdemo.html and http://www.swogstat.org/stat/public/binomial_calculator.htm.

The key practical rule is that, if no complications occur in an observational study of n observations or patients, the upper 95% confidence limit for the complication rate is approximately 3/n. Thus, despite no observed complications among the 102 patients in Fabregat-López's study, the serious complication rate with PLMA could be as high as 3%. Most anaesthetic complications are generally uncommon, with a prevalence below 1%. The commentary describes Table 1 as summarising sample sizes needed to observe at least one complication at various prevalences, strictly for single-group observational studies; comparisons with a control or test group require somewhat different, though related, statistics. The required samples are relatively large. The author argues that an individual or small group of anaesthetists cannot collect enough observations within a reasonable time to meaningfully address many relevant questions, and that collaborative, multicentre studies may be the only useful way forward for perhaps most important anaesthetic questions. Such collaboration is described as relatively rare in the specialty, although advocated elsewhere.

The ethical discussion begins with usual UK management of emergency appendicectomy: rapid sequence induction involving pre-oxygenation, cricoid pressure and prompt tracheal intubation, usually before manual ventilation of the lungs and facilitated by a rapidly acting muscle relaxant such as suxamethomium. This approach has historical authority as well as critics. Its components are intended to minimise aspiration risk—whether or not they actually do so—and to permit rapid rescue if tracheal intubation fails. Apart from pre-oxygenation, Fabregat-López and colleagues apparently omitted these components, something the commentary suggests could provoke debate independently of their choice of airway device.

The researchers nevertheless had appropriate ethics committee approval. The commentary says journals have a separate duty to ensure ethical standards. In defence of their approach, the researchers said it was normal practice in Spain, patients were fully informed of the research risks, and the team had considerable experience with the method. Publication therefore supplies professional opinion supporting PLMA use in emergency appendicectomy, subject to the original paper's precautions and limitations. The stated scope is appendicectomy, and possibly other lower abdominal operations—not emergency Caesarean section or upper abdominal surgery. UK researchers could cite the study in ethics applications for similar, related work.

The author cautions, however, that ethics committees and journals must also consider whether a study's results will be sufficiently robust to justify exposing patients to risk. Aspiration is described as possibly the most serious risk of PLMA use in emergency surgery. If aspiration risk with tracheal intubation and conventional rapid sequence induction is probably no more than 1:1000, and probably nearer 1:4000, a future study would need at least 3000 patients, ideally more than 5000, to examine this complication usefully. The author doubts that a smaller study would be ethically appropriate, because it would expose patients to risk with little likelihood of excluding or delineating the main concern.

The commentary connects this ethical problem to the history of probability. It says Poisson partly developed his distribution to address the martingale, a betting strategy involving repeated coin tosses that was popular in 18th–19th century France, although he believed all gambling—even an apparently fair game—was ruinous. His work built on Pascal's fundamental probability theory. Pascal also associated with gamblers, but, according to the account, experienced an epiphany after surviving a horse carriage accident and subsequently devoted his life to religion. Both mathematicians thus confronted the moral and ethical dimensions of probability calculations.

Returning to the small PLMA study, the author characterises its limited sample as a gamble that happened to produce no observed complications. In the author's view, even one aspiration event would have implied an unacceptably high aspiration prevalence of approximately 1%, potentially ending support for PLMA use in emergency surgery on statistical grounds. The commentary concludes that the study does not establish the true prevalence, but indicates that the described practice is commonplace. It had appropriate ethical approval, but did not confirm that PLMA use in emergency appendicectomy is appropriate practice without significant risk. It offers some—not a lot of—reassurance that aspiration risk is not prohibitive when PLMA is used as a rescue device, for example after failed tracheal intubation during rapid sequence induction.

The text acknowledges helpful manuscript comments from Dr Devinder Sivia, Senior Scientific Officer in the ISIS Data Analysis Group at Rutherford Appleton Laboratory and Lecturer at St John's College, Oxford. Its mathematical notes state N→∞, p→0, Np = constant. They clarify that, strictly, a binomial distribution concerns proportions in a population, such as disease presence or absence, or repeated yes/no events such as coin tosses; a Poisson distribution concerns events within a fixed time period. The formulae require further adaptation to account for variation and confidence intervals for prevalence.