Gallagher Re warns that the methods currently used to assess artificial intelligence (AI) models are not fit for underwriting purposes, and that better evaluation is needed to give insurers confidence in pricing AI risk. In a new standalone report, "Anthropic's Fourth Way: Why Restricted AI Models Are a Challenge for Insurers", the global reinsurance broking and advisory firm argues that current evaluation methods focus on performance rather than failure, and were not designed with underwriting in mind. "They indicate what a model can do under controlled, but insurers are concerned with how models fail, how often they fail, and whether those failures could be correlated across a portfolio," says Ed Pocock, global head of cyber security at Gallagher Re. The question matters for any insurer weighing AI exposure, including captives whose owners are deploying AI across their own business processes and considering how such risks might be underwritten or retained. Most AI models are presently evaluated using benchmarks, the standardised tests that score capability against fixed tasks. While useful for comparing models in controlled conditions, the report argues these do not fully capture how a model behaves under the ambiguous, unpredictable inputs it faces in deployment. A model that scores highly can still hallucinate, make inconsistent decisions, or fail in ways that are difficult to detect. Current methods also do not assess concentration risk, namely whether failures in widely shared foundation models could be correlated across multiple insureds. The report identifies benchmark contamination, where models are increasingly shaped by the tests used to evaluate them, as a further problem that can inflate published scores and reduce their value as a guide to real-world reliability. It also warns that where only a narrow set of behaviours is modelled, efforts to reduce failure and boost test performance can increase model homogeneity, compounding concentration risk. "This risks erasing useful differentiation between systems and increasing concentration risk," says Pocock. The report also examines the challenge posed by restricted-distribution AI models. Anthropic's Mythos model, released under its Project Glasswing programme, was made available only to a vetted group of partners rather than the broader market. Gallagher Re argues this represents a fourth category of frontier AI model, restricted distribution, sitting apart from open source, open weight and proprietary models. The distinction matters, the firm says, because it could set a precedent for the most capable models being kept from the independent evaluators that insurers and the wider market need to assess how they perform. The UK AI Security Institute has analysed Mythos and published its findings. However, Gallagher Re argues that, for insurers to price risk rather than uncertainty, restricted models need to be accessible to independent, third-party evaluators whose work can inform underwriting. "If a model cannot be independently evaluated, it cannot be meaningfully priced," says Pocock. "Insurers could end up loading for uncertainty rather than reflecting actual risk. That raises costs for everyone and slows the market's development." The report calls for a shift toward evaluation that tests AI systems as they operate: with real-world inputs, under adversarial conditions, and over time as models are updated. It identifies a need to measure hallucination rates, decision consistency, how models fail, and the potential for correlated failure across deployments. The firm points to signs of progress, including work by Epoch AI and Artificial Analysis on approaches that are harder to contaminate and more informative about failure. Gallagher Re argues the (re)insurance market can actively influence which models are deployed and how transparently they are evaluated through underwriting requirements, pricing signals and coverage design. "Better evaluation gives the market the tools to reward transparency and robustness," says Pocock. "Without it, we risk defaulting to scale and brand as proxies for safety, which could amplify the concentration risks we'll need to manage."
