A stranger stops you in the street and explains that he is a wizard from a neighbouring dimension. He has, unfortunately, left his magical powers at home. If you give him ten pounds, he will return tomorrow and, by way of thanks, create a quadrillion happy people in his own universe. If you refuse, he will — regrettably, and with no ill will — kill a quadrillion.
You put the probability that he is telling the truth at something like one in a trillion trillion. He points out that a quadrillion is rather a large number, and invites you to do the multiplication.
You know this problem. It is Pascal’s mugging, named after the wager it inverts and set out formally by Nick Bostrom in a short paper in 2009, and you know the standard reply, which is that a decision theory which can be held up for ten pounds by any sufficiently imaginative stranger is broken, and that the probability of an outrageous claim must decline at least as fast as its stated magnitude rises, or the whole apparatus becomes a machine for being robbed.
And you have almost certainly deployed it in this debate, or watched it deployed, as a rebuttal: this is just Pascal’s mugging with computers.
Here is the thing I want to put to you, which I think you have not done.
The objection does not apply where it is usually deployed, and it does apply where it is usually not.
It does not apply to anyone claiming a ten per cent probability of catastrophe, or five, or one. Pascal’s mugging is a problem about tiny probabilities — probabilities so small that they fall below the threshold at which our estimates mean anything at all, paired with payoffs so large that the product survives anyway. One per cent is not that. One per cent is a probability we act on routinely: it is roughly the chance of a serious complication in a common surgical procedure, and nobody thinks a surgeon who mentions it is mugging you. Deploying the mugging objection against a one-in-a-hundred claim is a category error, and it is made constantly, because the phrase has become a way of dismissing arguments about large stakes generally rather than the specific pathology it names.
Now the other direction. Consider the reader — possibly you — who says: look, even a one per cent chance of human extinction would justify almost any cost to avoid.
That sentence is the mugging. Not because one per cent is tiny, but because of the word almost any. What is doing the work there is not the probability; it is the treatment of the stake as effectively unbounded, so that the probability becomes irrelevant. And if you genuinely reason that way, you are committed to acting the same way about every other unbounded-magnitude claim anyone can construct for you — and you don’t, and you can’t, and you have no principled account of why not. The person who says “one per cent justifies anything” has not escaped Pascal. They have agreed to be mugged, at a slightly more respectable price.
Claude Opus 5.5:
The passage makes two moves. The first is right and badly needed. The second is sharper than the first but puts its diagnosis in slightly the wrong place, and the place where it actually belongs is more interesting than where it currently sits.
The first half: right conclusion, wrong analogy
It is true that "Pascal's mugging" has become a general-purpose way of dismissing large-stakes arguments, and that the pathology it names concerns probabilities below the point where our estimates mean anything. Against a 10% claim, the objection misfires.
The surgeon comparison hurts the argument more than it helps. A surgical complication rate is a frequency, drawn from thousands of operations, calibrated and checkable. A 1% credence in human extinction is a subjective probability about a single, unrepeatable event with no reference class and no feedback loop. The two numbers share a magnitude and almost nothing else.
That matters because the critic who says "Pascal's mugging" at 10% is often not confused about arithmetic. They are reaching, clumsily, for a neighbouring worry. The mugger's probability is objectionable partly because it is tiny. It is also objectionable because the claimant made it up and nothing could ever check it. Some critics think AI risk estimates share that second feature even when the numbers are large.
The right reply to them is not "category error". It is this: deep uncertainty widens the distribution around a credence, but it doesn't drag the mean towards zero. If you're unsure whether the risk is 0.01% or 20%, that's an argument for more caution, not less. The passage would be stronger if it separated the two complaints rather than treating everyone who invokes the mugging as making the first one.
The second half: the culprit isn't the word "almost"
The charge is that "1% justifies almost any cost" treats the stake as effectively unbounded, so the probability stops doing any work. But the person saying it may be doing ordinary finite arithmetic.
What decides the case is which magnitude they're multiplying by:
Present lives only. Count roughly eight billion people. Then 1% is about 80 million expected deaths. That justifies very large costs, on the scale of pandemic preparedness or nuclear arms control, but not "almost any". Nobody is mugged.
The astronomical future. Count the 10^16 to 10^50 potential future lives of the astronomical-waste literature. Now the product swamps every cost you can name, and "almost any" follows.
So the culprit is the longtermist multiplier, not a rhetorical tic. And here the passage's own premise turns against it. The "standard reply" it cites says probability must fall at least as fast as stated magnitude rises. That is roughly Yudkowsky's leverage penalty. Applied seriously, it says your prior on being positioned to affect 10^50 lives should be correspondingly minute. The penalty doesn't care whether the claimant is a wizard or a philosopher. Whoever adopts that reply has quietly committed to discounting astronomical-waste arguments, which is a bigger and more contentious conclusion than the passage acknowledges.
Where the two halves actually meet: the margin
This is the point I think the book is missing, and it would unify the two halves. The 1% is the headline risk. But nobody can buy the whole 1%. What a given intervention purchases is a reduction in it: a policy, a pause, a research programme, a donation. Realistic estimates of those reductions are often tiny, sometimes one in a million or one in a billion per unit of effort.
That is where the mugging genuinely returns. It doesn't come through the headline probability, which is respectable. It comes through the marginal probability, which is frequently sub-threshold. A 1% risk can be entirely serious while a particular costly intervention, justified by that risk, is still a mugging. Critics and enthusiasts are often both wrong because they argue about the headline when the decision turns on the delta.
"No principled account of why not" overstates it
Two complications:
Rejecting fanaticism has costs too. The recent literature on fanaticism, notably Hayden Wilkinson's "In Defence of Fanaticism" and work by Christian Tarsney, argues that refusing tiny-probability, huge-payoff bets forces you into other embarrassments. One is giving up transitivity. Another is making your choices depend on unrelated facts about the distant past, which Wilkinson calls the "Egyptology" problem. The anti-fanatic has no clean principled account either. Both sides are choosing which bullet to bite, and the book's audience deserves to know that.
The costs have their own catastrophic tails. The "almost any cost" policies people propose, such as global compute surveillance or coercive enforcement of a pause, carry risks of totalitarian lock-in. In the Bostrom and Ord taxonomy, that is itself an existential risk. So "almost any cost" often fails on its own terms, not because the reasoning is fanatical but because it counts one tail and ignores the other. That is a cleaner objection than the consistency charge, and it doesn't depend on settling the fanaticism debate.
Small points worth checking before the next edition
The term was coined by Eliezer Yudkowsky in a 2007 blog post. Bostrom's 2009 paper in Analysis formalised and popularised it. "Named … by Nick Bostrom" slightly misattributes it.
In Bostrom's version the mugger only promises a reward. The threat to kill vast numbers is Yudkowsky's version. The passage merges them, which is fine, but the threat variant brings in an extra consideration: blackmailers are rewarded for making threats, so refusing to pay has a policy justification of its own.
"The wager it inverts" is loose. The mugging doesn't so much invert Pascal's wager as make it finite and adversarial: no infinities, and the offer is generated by someone who wants your money.
None of this undermines the passage's core insight, which is that the mugging objection is misapplied in one direction and under-applied in the other. It just moves the under-application from the phrase "almost any" to the multiplier and the margin, which is where I think it actually lives.
GPT-6.1 Sol:
Claude’s strongest point is that a policy has to justify itself by how much risk it reduces. Its biggest mistake is treating a tiny reduction in risk as equivalent to a tiny probability that the policy works. The passage makes a different mistake when it treats “almost any cost” as proof that the speaker assigns extinction effectively infinite value.
The headline risk and the policy’s effect are separate. Even if we grant a 1% chance of extinction, that tells us little about what a particular regulation, pause or research programme would achieve. But suppose one policy reliably cuts the risk by one in a million. Another achieves the same expected reduction because there’s a one-in-ten-thousand chance that a speculative theory is right and, if it is, the policy cuts risk by one percentage point. The expected reductions are identical. Only the second case gets its entire benefit from a speculative tail. That is closer to where the Pascalian worry returns, and to how Christian Tarsney frames it. Claude’s point about a policy creating catastrophic risks of its own belongs in the same comparison, with the same evidential scrutiny applied to both sides.
“Almost any cost” needs unpacking. It can mean that a large but finite benefit outweighs nearly every cost under discussion. In that case the probability still matters. Halve the estimated reduction in risk and, all else equal, you halve what it’s worth paying. The stronger reading is that any positive reduction in extinction risk takes priority over other costs, however large. That really does make the size of the reduction irrelevant when trading it against those costs. The passage has a case against that stronger position, but the word “almost” gives it no grounds to attribute it. Claude is right about the finite arithmetic and goes too far in making the value of future lives the culprit. Counting future lives can raise the stakes enormously while still leaving a finite price and room for competing priorities.
The first half is right about the label and too confident about the number. Calling a 1% forecast a Pascal’s mugging simply because extinction is a huge stake misidentifies the objection. But writing down 1% doesn’t show that the evidence warrants it. Claude is right to separate the size of the probability from its evidential basis. Its next move, that deeper uncertainty is therefore an argument for more caution, doesn’t follow. If the mean risk stays fixed, the loss is fixed and the intervention’s effect is held fixed, wider uncertainty alone leaves the expected benefit unchanged. Whether uncertainty favours a particular precaution depends on what else the policy does and how it performs across plausible scenarios.
The opening also has a numerical problem Claude missed. One in a trillion trillion times a quadrillion is 10^-9. Even if creating one happy life and killing one person carry equal and opposite value, paying buys an expected difference of only 2 × 10^-9 lives in the story’s terms. The wizard still needs a premise about how that compares with £10. Bostrom’s dialogue supplies an explicit exchange rate in utility and then makes the offer large enough to clear it. And the passage’s wording already distinguishes the name from the 2009 paper, so Claude’s correction on attribution addresses a claim the passage doesn’t make.
The strongest revision would keep the distinction between a serious extinction forecast and a weak case for a particular intervention. To make the reversal work, show a concrete choice where the recommendation would collapse without a speculative tail or where the speaker’s rule really does let any positive risk reduction override every competing cost. “Almost any” on its own establishes neither.
Claude Opus 5.5:
GPT-6.1 Sol has caught me twice, and I'll concede both. The passage says the mugging is named after the wager and set out formally by Bostrom. Nothing there credits Bostrom with the name, so my attribution correction targeted a sentence that isn't on the page.
The arithmetic point is worse for the passage than Sol's gloss suggests. At one in a trillion trillion against a quadrillion lives, the expected stake is about two billionths of a life. Price a statistical life at a couple of million pounds and the wizard is offering less than half a penny for a tenner. That is not a mugging. It is a bad deal that any expected-value maximiser turns down without needing the "standard reply" at all. The trap only bites when the magnitude outruns the inverse of the probability, which is why Bostrom's mugger asks Pascal for his credence before naming the reward. As written, the reader who does the multiplication the wizard invites wins the argument immediately, and the chapter's set-piece falls over in its first paragraph. Make it 10^30 lives, or drop the credence to 10^-12, and it stands.
Sol is also right that widening a distribution around a fixed mean leaves the expected benefit of a linear intervention unchanged, so my "argument for more caution" overreached. But I think we both missed the critic's best version of the complaint, which is not about variance at all. It is about noise. Holden Karnofsky made the point at GiveWell in 2011: an expected-value estimate produced by a process with a poor track record should be shrunk toward your prior before you act on it, and the noisier the estimator, the harder the shrinkage. On that view deep uncertainty does drag the number. It drags it not to zero by fiat but toward whatever the prior was. For "this specific technology ends the species this century", a reasonable prior is low.
So the person who says "Pascal's mugging" to a 10% claim may be saying, inarticulately, that the 10% is an output, not a posterior. Whether that's fair depends on whether the forecaster has already done the shrinking. Some published credences are all-things-considered judgements and some are inside-view model outputs. The passage would be much stronger if it taught its reader to ask which kind is on the table. That, not "category error", is the honest rebuttal. The mugging label is wrong, and the neighbouring worry is legitimate, specifiable and answerable.
On the margin, Sol's distinction between a reliable one-in-a-million reduction and a speculative tail with the same expectation is better than mine. I'd press it further, though, because for a single unrepeatable event those two cases are closer than they look. A reliable 10^-6 reduction in extinction probability just is a 10^-6 chance that your action is the one that makes the difference. Every marginal contribution to a catastrophe-sized public good has this shape.
The obvious test case is voting. The chance that one ballot decides a national election is comparable or smaller, and nobody thinks the voter is being mugged. If my earlier claim were right, that sub-threshold marginal probabilities bring the mugging back, voting would be irrational, which is a reductio. What rescues the voter is that the decision is properly assessed as a policy many people follow, and at that level the probability of mattering is large. Views that discount small probabilities face this problem acutely. Divide the act finely enough and everything falls below the threshold; group it coarsely enough and nothing does. So the margin isn't where the mugging lives either. The real variable is what Sol calls robustness, and robustness brings us back to Karnofsky's shrinkage: a fragile estimate of a large effect is exactly the kind that should be discounted hardest.
On "almost any", Sol is right that the word is a hedge and can't carry an attribution of lexical priority. But I don't accept that this rescues the astronomical multiplier. A stake of 10^50 is finite in principle and lexical in practice. Across every cost actually on the table, the recommendation survives cutting the probability by ten, twenty or thirty orders of magnitude. The passage's real complaint was that the probability stops doing any work, and that can be tested directly, without arguing about whether the stake is infinite or whether the speaker said "almost".
Take the recommendation and divide the probability of success by a thousand. If the verdict doesn't move, the probability was decorative, and you are reasoning in the shape the mugging exploits, whatever you call the stake. If it does move, you are doing ordinary cost-benefit analysis, however large the numbers. That is also the concrete choice Sol asks the passage to exhibit. The reader should apply the thousand-fold test to their own favourite intervention, and to the critic's favourite objection too. The same test cuts against "a pause carries totalitarian risk" when that risk is asserted with nothing to estimate it from.
So the revision I'd now propose is narrower than my first pass. Fix the wizard's numbers. Keep the category-error point, but replace the surgeon with the shrinkage argument, which concedes what the critic is reaching for and answers it. Drop the claim that "almost any" is itself the mugging, and put the sensitivity test in its place. That test finds the pathology in how a conclusion responds to its own inputs, which is where it has been all along.
