A Difficulty for Promotions: How Often the Brain Should Change, Measured

Abstract

The fleet promoted one set of weights in a day. Paul asked for "something like a block height difficulty": a bar that lowers itself until promotions come at a set pace, "a promotion every 20 minutes at average", with read-on stages that grow by "more like 1.2x or 1.6x, instead of double".

The difficulty is built and live. Every bar the planner uses (each read-on stage, each pass, and the held-out reads' z) is lowered together by one number, the ease. The ease rises while no promotion comes and falls back when one comes early, as a block's difficulty does. The stages now grow 1.5×. After each promotion, an anchor read plays today's defaults against those from three promotions back, so a promotion that made things worse is undone by the same machinery that made it.

The pace of 20 minutes was not adopted. A simulation, fitted to what the fleet's own trials measure, shows why:

• at one promotion every ~36 minutes, seven in ten promotions are truly worse, and the brain loses about 3.7 seats a day;
• the most gain a day comes at one promotion every 4–5 hours;
• its floor, an ease of 1.5, keeps the truly worse to one in ten or fewer.

Three things were found on the way, and fixed:

1. The held-out reads threw away real finds (w1198, w1266): each read had to pass alone. They now pass pooled.
2. Every promotion since w0034 had failed for a reason that had nothing to do with its numbers. Main had failed its own tests since dc3df01, because the promotion committed weights.rs and not tactics.rs. Main is repaired, and the promotion commits both.
3. My first measure of the trials was biased, and the floor I first shipped (2) came from it. Section IV has the measurement and the correction.

I. Objective

1. Make the bar for a promotion adjust itself toward a target pace.
2. Make the read-on stages grow by about 1.5× instead of 2×.
3. Find the pace that makes the brain strongest, and say what 20 minutes would cost.
4. Guard against the promotions a lower bar lets through that are truly worse.

II. Materials

III. Method: the difficulty, the stages, the anchor

The ease. Every bar is lowered together by one number, in errors:

The held-out reads' z is the bar that matters most. The families gauntlet and the brain's tests are not lowered: they are pass or fail, not a measure.

How it adjusts. The rule (ease(), state difficulty):

• Rising. For one target interval after a promotion (4 hours, weights.promote_every_min), the ease holds. After that it rises by 1 for each further interval, up to the floor.
• Falling. A promotion that comes before its time lowers the ease by half a step for each halving: one that comes in half the time lowers it by 0.5. The step for falling is half the step for rising because one quick promotion is as much luck as a sign. A first version used a full step, and one promotion 36 minutes after another sent the ease from its floor straight to 0.
• From now. The first ease began when the change shipped, not from the last promotion a day earlier. So it eases in over hours rather than jumping to the floor.

This is a block's difficulty turned around. A block chain lowers the work when blocks come slowly. Here the bar lowers when promotions come slowly, and rises again when they come fast.

The stages. A trial is read on until it is decided (Fishtest's sequential test, as in 12347ca). The stages now grow 1.5× instead of 2×:

(Each bar less the ease.) Trials already partway through moved onto the nearest stage. A few of them got one small top-up read of 14 seeds.

The held-out reads, pooled. Before this, each held-out read of 400 seeds had to pass alone, at about +0.55. A trial truly +0.6 better cleared both about one time in three. Pooled, the two reads are one read of 800 seeds (held_pass). Against a trial no better, the pooled rule at z 3 is about as strict as the old one (a false pass one time in 750, against one in 2,000). A trial +0.6 passes six times in ten instead of one in three.

The anchor. A lower bar lets some truly worse promotions through. Once a promotion is built on its machines, anchor_trial() takes the defaults from three promotions back and queues them as an ordinary trial, with its census and two held-out reads. If today's defaults are better, it fails. If not, it passes like any other trial and promotes itself back. No extra code path decides a revert: the revert is a promotion, with the same tests and families. The first anchor, w1374, plays the defaults from before w0034 against today's.

IV. Measuring the trials

Everything in Section V depends on one input: how the trials' true gains are spread. That input was measured three times, and the first two were wrong.

First, the dropped trials only: N(−0.54, 0.22), biased. I took the trials read only once, at 55 seeds. Those are the ones the read on dropped, which are the worst. A narrow, low prior makes every bar look worse than it is. I had also rounded this to N(−0.45, 0.30) for the first table I showed, which happened to land near the truth by accident, and the floor of 2 came from that table.

Second, every trial's first census: N(−0.32, 0.44), too wide. Each trial's own first census, whatever came after:

The reads spread by 0.81 against an error of 0.68, which leaves 0.44 for the true gains. Under that prior, the strict rule should have found a promotion every 87 minutes. The fleet found two in a day, so this prior is too wide. Part of the excess spread is noise the census's own error does not count.

Third, held to the held-out reads: a bulk N(−0.35, 0.24) and 0.3% real finds at +0.6. The held-out reads are fresh seeds, so they show what a first read was really worth. The trials whose first read came out near z 2.3 averaged −0.10 held out. With an error of 0.68, that shrinkage implies a true spread of about 0.24. A normal that narrow almost never produces a trial like w1198 (+0.67) or w1266 (+0.59), so the prior needs a thin tail of real finds as well. With 0.3% of trials at +0.6, the simulation finds one promotion every 16 hours at ease 0, which matches the fleet's record of about two a day. This is the prior Section V uses. The other two are kept as the edges of what the data allows.

V. Results

What each ease buys. From difficulty-sim.py, with the bulk N(−0.35, 0.24), 0.3% finds at +0.6, a seed's swing of 4.9 and 5,250 seeds an hour:

Twenty minutes is past what the trials hold. Even at an ease of 4, the fleet promotes only every 36 minutes. By then seven promotions in ten are truly worse, and the brain loses 3.7 seats a day. A faster fleet promotes sooner at the same ease, but the truly-worse share is set by the bar, not by the fleet.

The floor, against each prior. Gain a day:

No one floor is best under every prior. Under three of the four, 1.5 comes within 0.2 a day of the best gain; under the narrow prior with no finds it reaches 72% of it. It keeps the truly worse at or below one in ten under all four. A floor of 2 does as well on gain only under the narrow prior, and lets 11–21% of promotions through truly worse. The floor is 1.5, and the target of 4 hours is the floor's own pace, so the ease rests below the floor rather than sitting pinned to it.

The held-out reads, pooled. Of the 37 trials on today's defaults with both held-out reads:

The old rule lost w1198 and w1266. They were the first two trials to pass the sequential read, which shows that the read on works. Their families held (job f0029).

Why neither promoted: main was red. The auto promotions of w1198 (r0046) and w1266 (r0048) both failed at the brain's tests, and two failures in a row paused auto mode. The tests on origin/main alone:

weights::tests::default_weights_are_todays_constants ... FAILED
test result: FAILED. 108 passed; 1 failed

set-defaults.py writes each weight to weights.rs and, where a field shares its name, to Tactics::default() in tactics.rs. The promotion command ran git add source/treadcore-brain/src/weights.rs alone, so w0034's commit dc3df01 changed one file of the two. Main has failed this test since, and every promotion after would have failed it too, whatever its numbers.

• The repair. Treadcore d7e8c09 sets Tactics::default() to w0034's values (w_near 2.0, w_in_range 1.4, kite_in 4.5, longshot_every 2.5). All 109 tests pass. Tactics::default() is used only by the tests, and the game reads the weights, so nothing plays differently, and every census on today's defaults stays valid.
• The promotion now adds both files.
• The pause (same_failure) now waits for a person only when two failures share a failing test, and main has not moved since the later one. The case it exists for, something every candidate trips on, still pauses. A candidate that fails on its own numbers is rejected, and the next is tried.

w1198 also failed the_square_law (its engage_odds 0.175 changes a decision that test pins). Both w1198 and w1266 failed a_capped_castle_fills_then_lines_the_road (both move post_spacing). Those tests pass on main, so these two candidates are rejected on their own numbers. They remain in auto_tried.

The controller. Ease against time with no promotion, then a late promotion, then an early one (floor 2 in this run, before the floor changed; the arithmetic is the same):

Live. After the swap, the planner ran on the live queue with no error. Health reads: "difficulty: bars lowered 0.0 errors (held-out z 3.0), 9 min since it began, aiming at one each 4 h 00 m". The anchor trial w1374 is queued. On a copy of the queue set to the old floor of 2, six trials on today's defaults became candidates and 16 more qualified for held-out reads. That is the same rush the live queue will see in smaller steps as the ease rises.

VI. Discussion

Why a target pace is the wrong thing to fix. A block chain can set any pace it likes, because every block is valid by construction. A promotion is a claim that a change is better, and the claim is a measurement. Lowering the bar buys promotions, but not improvements: past the peak, each extra promotion is more likely noise than gain. The difficulty is still worth having, because the peak is far from where the fleet was. The strict bar made one promotion a day, at +0.9 a day; the floor makes about six, at about +1.8. What it should aim at is the gain a day, and the pace follows from that.

A measurement is only as good as what it is held to. The first prior came from a convenient subset, and it was wrong. The second came from all the data, and it was wrong in the other direction, because the census undercounts its own noise. Only the held-out reads, fresh seeds read after the decision, could say what a first read was worth. The same lesson explains the winner's curse the sequential read was built against: a trial's first read is not its true value.

What the simulation leaves out.

• Each promotion makes every trial in flight a read against old defaults. The planner drops those reads (same_defaults), and that cost is not simulated. A trial's whole read on is 418 seeds, about five minutes of the whole fleet, so at one promotion every four hours few are cut short.
• After a promotion the trials are drawn around new defaults, so the prior moves. difficulty-sim.py measures it again from the queue each time it is run.
• Each trial's gain is treated as independent. Trials near one fit are not, and the "best" repeats are the same point read again.

More machines. Promotions come at a rate proportional to the seeds an hour, so a machine that adds 8% of the rate adds about 8% of the gain a day at the same ease. It does not change which ease is best.

Still open.

• Settled the same day: a_capped_castle_fills_then_lines_the_road pinned a constant. road_posts lines only a road longer than twice post_spacing, and the test's second depot is the nearest one past 30 tiles. So at a spacing of 28 (w1266) or 40 (w1198) there was no road to line, and the test failed on a number it does not test. It now pins post_spacing at 10, beside the castle weights it already pinned (treadcore 9032026). With w1266's values, all 109 tests pass. A candidate whose tests failed is now tried once more when main moves (retry_after_main).
• Settled too: the_square_law pinned a constant as well. Its rule is relative to the bar: below engage_odds, with teammates on the way, a bot masses. But it asserted that 1 against 2 (odds 0.25) masses, which is true only while the default engage_odds sits above 0.25. w1198's 0.175 failed it. It now pins the bars it was written for, 0.7 and 2.0 (treadcore 5ef8ad7). All 109 tests pass with w1198's values.
• The anchor reads catch a drift only once it is three promotions deep.

VII. What to watch

• Health's difficulty line shows the ease and the time since the last promotion. It turns amber at the floor with no promotion in twice the target time, which means the trials hold nothing at any bar a promotion should pass.
• The anchor trials, titled "anchor: the defaults before rNNNN". One that passes means a promotion is being undone.
• difficulty-sim.py QUEUE measures the prior from the queue and prints the table again. Run it after a few promotions. If the gain a day peaks at another ease, change EASE_MAX and PROMOTE_EVERY together.

VIII. Addendum, the same evening

With the brain's tests set right and w1266 promoted (e669921), Paul set the aim at one promotion every 2.5 hours. That is the pace of ease 2 under the measured trials, so PROMOTE_EVERY is 150 and EASE_MAX is 2.0. The cost is in Section V: there, about one promotion in five may be truly worse, against one in fourteen at 1.5. The anchor reads are what stand against that. In a new generation, only what was found carries over: the search moves to the new commit at once, and the older finds are read again on it. Builds of commits main has already moved past are no longer built (a busy calber had three queued).

References

[1] Fishtest, "Stockfish testing framework: SPRT and the pentanomial model," official-stockfish/fishtest wiki.

[2] A. Wald, Sequential Analysis. New York: Wiley, 1947.

[3] S. Nakamoto, "Bitcoin: A Peer-to-Peer Electronic Cash System," 2008, §4 (proof-of-work difficulty).

[4] B. Efron, Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. Cambridge University Press, 2010 (shrinkage, and the winner's curse).

[5] orrery docs/fleet-train.md, "A trial is read on until it is decided" and "A difficulty, like a block's."