A city discovers that neighbourhoods receiving the most road repairs also report the most accidents. The association invites two incompatible headlines: repairs are dangerous, or repairs are being directed towards dangerous roads. Both are compatible with the observed pattern because the data record where intervention occurred, not what would have happened in those same places without it. Adding more records can estimate the association more precisely while leaving this ambiguity untouched. The fundamental shortage is not necessarily a shortage of observations. It is a shortage of information about the process that generated the comparison.
Causal reasoning concerns interventions: how an outcome would differ if some condition were changed while relevant background features were appropriately accounted for. Prediction asks a different question: given what is observed, which outcome is likely? A variable can predict an outcome without offering an effective means of changing it. The arrival of repair crews may predict a dangerous street because crews are dispatched in response to danger. Preventing their arrival would not remove the underlying danger. A predictive relation becomes misleading when transported into an intervention without examining why the relation existed in the first place.
The difficulty can be expressed through counterfactual outcomes. For a particular road at a particular time, we observe what happened under the policy actually implemented. We do not simultaneously observe what would have happened to that identical road under an alternative policy. Comparisons with other roads attempt to supply the missing information. Their credibility depends on whether the comparison roads differ in ways that also affect accidents. This is why a large sample does not automatically solve confounding: repeated observation of systematically different groups can reinforce a biased comparison with extraordinary numerical confidence.
Suppose the city adjusts for traffic volume and road width. The adjusted comparison may be more informative if these variables help explain both repair allocation and accident risk. But adjustment is not a ceremonial purification achieved by including every available column. Some variables are consequences of the intervention; others are consequences of several different causes. Controlling for the wrong variable can block part of the effect of interest or create an association that was absent before adjustment. The statistical operation acquires its meaning from a causal account of the variable's role, not from the mere availability of a measurement.
Consider a deliberately simplified admissions example. A programme admits applicants who have either exceptional examination performance or exceptional artistic achievement. Among those admitted, lower examination performance may tend to accompany stronger artistic achievement, even if the two attributes were unrelated in the original applicant population. Selection has manufactured a relationship: within the selected group, weakness on one admission route must often be compensated by strength on the other. Admission is a collider, a shared consequence of the two attributes. Restricting attention to admitted applicants can therefore distort the relation one intended to investigate.
The example reveals why representativeness involves more than demographic variety. A dataset can contain people from many backgrounds while remaining selected through a mechanism that alters relationships among variables. Complaints recorded by an institution reflect both underlying problems and willingness or ability to complain. Analysing only complaints may confuse severity with access to the reporting channel. Expanding the database of complaints improves coverage within that channel without necessarily revealing the people whose circumstances prevent reporting. Missingness can be structured by the phenomenon under investigation rather than scattered harmlessly around it.
Random assignment helps by making treatment allocation independent of prior characteristics in expectation. It does not guarantee perfect balance in every realised experiment, nor does it automatically solve non-compliance, attrition, measurement error, or the transfer of findings to different populations. Its power comes from a specified allocation mechanism, not from a magical status attached to the word random. A carefully designed experiment answers a particular question under particular conditions. Asking it to settle every policy question can be as careless as dismissing observational evidence simply because randomisation was unavailable.
There are circumstances in which the intervention itself lacks a sufficiently precise definition. 'Improve education' could mean smaller classes, different assessment, better nutrition, or a change in teacher training. These versions may produce different outcomes through different pathways. If they are combined under one treatment label, the average effect may refer to an unstable mixture. A policy recommendation then equivocates between the category analysed and the action proposed. Before estimating whether a treatment works, one must specify what counts as receiving it. Conceptual precision is part of identification, not a literary ornament added after calculation.
Time introduces further complications. A road repair may initially disrupt traffic, subsequently reduce accidents, and eventually attract additional vehicles. Evaluating only the first week yields a different object from evaluating several years. Neither horizon is intrinsically correct; each answers a different question. Similarly, a policy can change the conditions on which its original evaluation depended. If safer roads induce more travel, the number of accidents and the risk per journey may move differently. An apparent contradiction can arise because one observer measures total burden while another measures individual exposure. Their disagreement may concern denominators rather than arithmetic.
The attraction of automated analysis lies partly in its capacity to postpone these difficult choices. A model can rank variables, discover interactions, and produce predictions before anyone agrees on the intervention or the target population. Such work can be useful, especially for generating hypotheses. But an impressive fit does not confer causal authority. When a system recommends action, the burden shifts from reproducing patterns to explaining why those patterns should survive deliberate change. The distinction becomes politically consequential when decisions affect people who cannot inspect or contest the assumptions hidden inside a technical recommendation.
A responsible analysis therefore separates what the observations show from what the causal interpretation requires. It states the comparison, the relevant assumptions, and the conditions under which the conclusion might fail. Sensitivity analysis can explore how unmeasured influences would need to behave to alter the result, without pretending to observe those influences directly. Different designs can be compared to see whether their distinct weaknesses point towards similar conclusions. This triangulation strengthens a judgement through diversity of evidence, not through repetition of the same vulnerable comparison across ever larger quantities of data.
The city should consequently welcome its expanded records while resisting the belief that expansion has abolished interpretation. Data can narrow uncertainty only relative to a question and a model of how observations arose. Where that model is mistaken, precision may amplify error instead of correcting it. The mature promise of quantitative reasoning is not that numbers relieve institutions of explanation. It is that clearly specified assumptions and comparisons make explanations more accountable. More data can improve the answer, but only after someone has established which answer the data are actually capable of supporting.
Source notes
- [1] Hernan and Robins, Causal Inference: What If (2020; author-hosted version). Background: causal comparisons, confounding and selection. Open primary source