What is each chess mistake worth?

Placebo:

Nothing is fixed: both players are the same

Hold the win:

Half of the costly moves made when ahead

Punish it:

Half of the costly moves that leave an opponent's gift untaken

Threat check:

Half of the costly moves that run into a mate, a hanging piece or a tactic

Opening:

Half of the costly moves in the first ten moves

Endgame:

Half of the costly moves in the endgame

Small slips:

Half of the moves that cost under 10 points

After a blunder:

Half of the costly moves straight after the player's own blunder

• ›Three fixes were worth the most at every level: Threat check (107 to 140), Small slips (95 to 131) and Hold the win (82 to 128).
• ›Punish it falls as players get stronger: 67 at about 900, then 41, 49, 16 and 13. At the top two levels its interval includes zero.
• ›The opening pays most at the bottom: 70 and 53 at about 900 and 1300, then 16, 0 and 27.
• ›The endgame pays most at the top: 45, 29, 29, 50 and 72.
• ›Playing better straight after a blunder was worth least: 26, 12, 34, 3 and 1. Few moves qualify, at most 0.3 a game.
• ›Our study of real games found small slips barely change with rating, and we left them out of the plan's leaks. They do not separate the levels, yet cutting them here paid as much as anything. What separates players and what would help one player are different questions.

• ✓ Held

  Placebo: two identical players score even at every level. If not, the simulation has a bug.

  Every level's interval includes zero. Over all 5,000 placebo games the result was +3 Elo (−6 to +13).

• ✕ Failed

  Threat check is worth more than Small slips at every level.

  After correcting for seven tests, this held only at about 900. Small slips came out level or ahead at about 1700 and about 2500.

• ✓ Held

  Punish it is worth more at about 900 than at about 2500.

  67 against 13. Our simulated 2500s are given too few gifts (4 points against a real 10), so the true drop is probably smaller.

• ✕ Failed

  Hold the win is worth more at about 900 than at about 2500.

  111 against 95, a difference within the noise.

• – Not contradicted

  A point of winning chances buys the same Elo whichever kind of mistake it comes from.

  Not rejected at any level (p from 0.14 to 0.54), so the plan can keep ranking leaks by points for now. The test is weak: see the note on small slips below.

• ~ Mostly

  An expectation: fixes rank by how many points they remove.

  The lower ranks matched almost exactly. Small slips removed the most points at every level but ranked first by Elo at only two of the five.

Looked at afterwards

None of this was planned in advance. We chose these cuts after seeing the results, so they can suggest and cannot establish.

• One corrected move a game was worth far more for costly errors than for slips: Threat check 112 to 146 Elo, Hold the win 131 to 184, Small slips 11 to 16. Small slips needed 6 to 10 corrections a game for their total; Threat check about one. Whether one correction is as hard to learn as another, this trial cannot say.
• A point lost to a small slip was worth 52 to 62% of a point lost to a threat, at every level. Some of that may be engine noise: these games were read at depth 8, and a slip that small is partly the engine changing its mind.
• A stronger simulated opponent did not draw more errors. Facing the level above, the weaker player gave away 4 to 5% less, not more. Real strength probably includes making a position hard for the other player, which these players cannot do. That may be why the ladder below came out so short. We have not tested it on real games.

How it was done

Five levels: about 900, 1300, 1700, 2100 and 2500. At each, a player built on Stockfish 19 was tuned on its own games, with seeds the trial did not use, to give away what real players of that level give away, using 3,412 real Lichess games as the target. Then 1,000 games for each of eight conditions at each level: a player with half of one kind of move replaced by Stockfish's choice, against an otherwise identical player, with colours drawn at random. The placebo changes nothing and checks the simulation. A further 1,600 games between neighbouring levels check the calibration. The analysis ran once, after the last game, and nobody looked at a result before then.

How far to trust it

• The ladder failed. Neighbouring levels are 400 rating points apart in real play. Here the upper level won by 109, 1, 72 and 272 Elo: 454 in all, against a real 1,600. The simulated 1700 was no stronger than the simulated 1300 when they met. We had planned to scale results by this ratio, but the ratios run from about 1.5 to over 400, so we do not rely on it. Treat the Elo here as a way to compare fixes, not as rating points.
• Match to real players. Total points given away per 40 moves came within 6% at all five levels (227 against 229, 182 against 188, 171 against 173, 145 against 137, 102 against 108). Some categories miss: the endgame at about 1300 (20 against 42), gifts at about 2500 (4 against 10), the opening at about 2500 (23 against 11), hanging pieces and mates at about 2100 (24 against 13).
• Did each fix do its job? Against the placebo games, Threat check, Punish it and Small slips cut their own number by 45 to 57%, as designed. Hold the win, Opening and Endgame cut theirs by 10 to 51%, because those numbers include small moves they do not touch. At about 2500, Hold the win cut its number by only 11% and was still worth 95 Elo: a player who holds on stays ahead longer, with more moves on which to lose a little.
• Could a player see the change in their own numbers? For a cut of one half: Small slips in 2 to 4 games, Threat check in 12 to 30, Punish it in 44 to 127, Endgame in 66 to 284, Opening in 109 to 1,859, and Hold the win in 74 to over 1,000, or not at all at about 2500. These are optimistic. A cut of a tenth would take about 25 times as many games.
• Games and draws. Simulated games ran 55 to 95 plies against a real 63 to 81, and 1 to 8% were drawn against a real 2 to 7%. Fixes are an extreme treatment: half of a kind of move becomes Stockfish's best, which no training does.

The hypotheses, sample sizes and analysis were fixed in advance and fingerprinted (SHA-256 6eff027be831d0e711bc33916111f4cbcd30a8da5c1e52bc77e621b099a837f3); any change is listed with its reason. Stockfish 19. Games from database.lichess.org, CC0.