Eleven machines · one queue · one brain

Orrery

The fleet that trains treadcore's bots, and the console that runs it. A planner guesses a number, the machines read it, and what holds goes into the game.

MIT · Rust, no crates · a brushed-metal console on macOS · Linux and Copal nodes · CUDA islands

Orrery's Pipeline Board at 10:05 on 5 November: the line from the next trial to the game's brain as a Gantt chart with its machines and expected times, and below it the tasks as a kanban of waiting, running and done
05:46 the keeper restarted on new code: 11 jobs handed over, none run again07:54 r0186: grove/w10040 promoted onto the line, 167 tests, every family, the gauntlet08:27 run r0188 finished: bio, 400 seeds08:27 grove/w10689 refused by the trade rule: it gives up fights10:00 r0187 handed back to the planner, step 16 of 3010:05 busy, last 6 h: CPUs 95 to 98%, GPUs 100%the GPU islands keep six segments ahead, on purpose

The line

From a guess to a commit.

  • A first read plays a trial against the game's defaults on 55 seeds, and drops the hopeless for the price.
  • Read on, then held out. More seeds until it is decided; then 400 no search has seen, the only fair read of a winner.
  • Families, gauntlet, tests, and onto the line; the line goes to main when it reads past it.

Auto mode promotes by itself. A person can hold any dial.

The Flow page: every stage of the line as a box with what it takes and gives, and one real trial's reads traced through
The Flow: each stage, what it takes and gives, and one trial's numbers as they moved.

The machines

Eleven lanes, ninety-eight percent busy.

  • Eight CPU machines play the census, three thousand jobs a day, the search's first reads and read-ons most of it.
  • Three GPU cards evolve crucible's islands, a population per card, a segment every twelve minutes.
  • The keeper hands each free lane its next job, and keeps its running jobs across its own restart.

Left of now is measured; right of now is a forecast, and says so.

The Timeline: six hours of each machine, measured, then the forecast from medians
Six hours of each machine: the search in blue and green, held-out reads in orange, the GPU rows one unbroken bar.

The round table

Who waits on whom. And who helps.

  • Red between two machines: work standing while another finishes. Today there is none.
  • Amber to the table: a run handed back to the planner, until its next pass.
  • Green back to the queue: the small returnable reads each machine takes when its own line waits.

The machines do not wait. The work does, and the page says where.

The Blame page: eleven machines round a table, the queue in the middle, green arrows of help, an amber hand-back, loops of GPU segments waiting their turn
The Blame page, live: no machine waits on another; r0187 is handed back to the planner.

The brain, read

What the generations found.

  • Terrain: every dataset's surface and its local maxima, the dials the search climbs.
  • Matrices and spectrum of every learned net, generation by generation.
  • Candidates, nets, recordings, films: every result has a page and a source.

Hover almost anything: the inspector says what it is and where it came from.

Results, Algorhythm, Terrain: a dataset's surface with its local maxima marked
Results, Algorhythm: the terrain the search climbs, and where its maxima are.

This morning

Read from the queue's own records at 10:00 on 5 November; the method is in The Duty Cycle.

MeasureValueNote
CPU lanes busy99.3%eight machines, since the low water counts only CPU work (was 97.7% over 12 h)
GPU lanes busy, last 12 h99.8%three cards, crucible's islands
Jobs a day3,9551,519 first reads, 1,505 read-ons, 378 GPU segments, 120 held-out reads
Generations, last 7 days27a median 2.2 hours apart
The search's funnel, a day1,520 → 3652 read on, 75 held out, 3 generations
Jobs run again since the handover0the keeper keeps its jobs across its own restart
A transform step, end to end2.3 minwas 7.6: the keeper asks the planner when a chain's step returns, and a pass takes 43 s, was 105