'The model correctly identified the districts where demand would grow,' said the planning analyst, Rohan. 'And we invested in those districts after seeing the forecast,' replied his colleague Farah. 'How much of the growth followed the investment?' They were evaluating a hypothetical public transport expansion. Rohan regarded the forecast's success as evidence of its accuracy. Farah thought the conclusion incomplete. A prediction used to allocate resources could help produce the outcome it predicted, an instance of reflexivity. Agreement between forecast and observation might therefore reflect foresight, intervention, or some combination. The model had entered the process it was supposed to describe.
Rohan asked whether she was suggesting that successful predictions were inherently suspicious. 'No,' she said. 'I am asking which claim success supports.' If the forecast aimed to describe demand under unchanged policy, investment altered the conditions of evaluation. If it aimed to predict demand after a specified investment, the intervention belonged inside the question. The distinction was not semantic fussiness. Two apparently identical forecasts could refer to different counterfactual worlds. Before comparing prediction with reality, they needed to state which world the model had been intended to predict and which actions had been taken because of it.
He pointed to historical validation. The model had performed well on earlier data. Farah asked whether those data reflected similar allocation rules. If past planners invested where they expected demand, observed demand already incorporated prior judgements. A new model might learn the traces of those decisions and reproduce them as though they were independent properties of districts. 'We may be predicting the history of investment as much as latent need,' she said. Rohan conceded the possibility but noted that historical patterns remained useful. 'Useful for what?' she asked. 'Forecasting continuation is different from identifying where a change would produce the greatest benefit.'
They examined a district with low recorded ridership. Rohan considered it a weak candidate for expansion. Farah observed that its service was infrequent and poorly connected. Low use could indicate low demand, but it could also indicate a service inadequate to reveal demand. If investment followed only observed use, the district might remain underserved indefinitely. 'The measure records behaviour under the options provided,' she said. 'It does not directly reveal what people would choose under options they have never had.' The allocation rule could then preserve the conditions from which the model inferred that further allocation was unnecessary.
Rohan described the concern as a feedback loop. Farah agreed but emphasised that this performativity, whereby a forecast helped bring about its object, was not automatically harmful. Investment in a promising district might generate valuable activity, making the forecast a useful component of planning. The difficulty was identifying the objective. If the purpose was to maximise near-term ridership, reinforcing existing demand could be defensible. If the purpose included access, inclusion, or the development of neglected areas, the same allocation might perform poorly. 'A model can be accurate about an outcome that is an inadequate proxy for the public goal. Technical validation cannot choose the goal on our behalf.'
He proposed a fairness constraint requiring equal investment across districts. Farah resisted the apparent simplicity. Equal expenditure could produce unequal access because population, geography, and existing infrastructure differed. A district with greater need might require more resources to reach a comparable service level. Conversely, pouring money into an ineffective design would not become fair merely because the district was disadvantaged. They needed a substantive account of the outcome they sought and evidence about how spending affected it. Equality of input was one possible rule, not an exhaustive definition of equitable provision.
'How can we evaluate the alternatives without knowing the future?' Rohan asked. Farah suggested limited pilots, phased implementation, and comparisons designed to distinguish treatment effects from pre-existing trends. None guaranteed certainty, and experiments could raise practical concerns when services were urgently needed. But the absence of a perfect design did not justify treating the existing allocation as an experiment that had already answered every question. 'We can learn from variation if we understand how it arose,' she said. 'What we cannot do is ignore our intervention and then congratulate the model for predicting a world partly organised around its recommendation.'
Rohan worried that exploration would sacrifice resources on areas expected to perform poorly. Farah acknowledged the trade-off. Exploitation used current knowledge to pursue apparently favourable options; exploration generated information about options current knowledge might undervalue. An allocation system devoted entirely to exploitation could become confident without learning about excluded alternatives. Yet indiscriminate experimentation could also waste resources. The balance depended on stakes, uncertainty, reversibility, and the value of future learning. The problem was not solved by selecting either caution or curiosity as an absolute virtue. It required deciding where additional evidence was worth its cost.
They then considered accountability. If the model recommended a district and the outcome disappointed, officials might blame the model. If the outcome succeeded, they might claim credit for decisive investment. Such asymmetry could distort both deployment and evaluation. Rohan suggested documenting the recommendation, the reasons for accepting or rejecting it, and the subsequent intervention. Farah agreed. An auditable decision record would help separate model output from human discretion and implementation. It would not eliminate responsibility's complexity, but it could prevent the complexity from becoming a convenient refuge in which every participant denied ownership of an unfavourable result.
The discussion also altered their view of transparency. Publishing a list of model variables would not be enough if the decisive assumptions concerned how transport supply generated recorded demand. A technically detailed explanation could remain substantively opaque. Citizens needed to understand what objective was being optimised, which outcomes were observed, and what possibilities the data could not reveal. They also needed a route to challenge errors and overlooked needs. Transparency mattered because it enabled contestation, not because displaying complexity automatically made a decision legitimate. A visible formula could still conceal an unexamined choice of purpose.
Rohan rewrote the evaluation's opening. The model had identified districts that subsequently experienced growth, but the evaluation could not attribute that agreement entirely to predictive skill because investment had followed the recommendations. He added a proposal for testing alternative allocations and measuring access alongside ridership. Farah thought the revised account less celebratory and more useful. It distinguished what they knew from what they needed to learn. The result did not discredit the model. It located the model within an institution whose actions changed the evidence on which future models would be trained.
As they closed the report, Rohan asked whether a forecast could ever stand outside the world it described. Sometimes its influence was negligible, Farah said; sometimes it was decisive. The important task was to investigate the relation rather than assume independence. A map used to build roads would eventually resemble the territory for reasons different from a map that merely recorded existing roads. Their transport model might be both map and instruction. Evaluating it responsibly meant asking not only whether the future resembled its forecast, but also what kind of future its authority had helped to make available.
Source notes
- [13] Perdomo and colleagues, Performative Prediction, ICML (2020). Background: predictions can influence the outcomes they predict; the transport dialogue is original. Open primary source