Abstract
The parts:
- The engine, orrery's
tools/fleet-queue.py: a queue, a lane per machine, and a keeper that keeps them running and stops adding after five failures in a row. - The planner, treadcore's
scripts/fleet-plan.py, which fills the queue with trials of the scripted brain's 110 weights and rounds of the learned net, in turn. - The controls: stop adding, and collect results and stop now. They are commands, and buttons in orrery's Training view.
The sample run, on calber, columbia, nasser and the Mac:
- It built the queue's commit everywhere.
- It ran a net round: trained on calber's GPU, judged on columbia, candidate +0.64 ± 0.40 over 4 seeds, so the best stays.
- It ran 70 weight trials in five minutes. One read past two errors and was queued for its held-out reads.
- It found two faults on the way.
I. Introduction
A. Intent
Have orrery keep the fleet busy training treadcore, rotating through every variable, with ways to stop that keep the results.
B. The prompts
can the trainer for treadcore train all the variables, and maybe some feedback placeholders for positive and negative feedback . unsure if its a good idea, is this the right type of network. the orrery should have a treadcore trainer in a way such that It knows to train all the variables and rotate between them. Placing items onto the queue, and keeping these system alive, while allowing a stop adding to the queue button. and a collect results and stop now button.
take pictures of orrery and show detailed descriptions of why the picutres of the net look the way they do. describe the user interface and the improvements it would need to show analysis of AI training.
impressive lab report . place in documentation . can we get a sample run of orrery filling up its queue. lets see it work . tell me how to use it.
Paul's choices: both kinds of variable in one queue; reward terms as the feedback placeholders.
II. Decision Log
| # | Cause | Decision | Not taken, and why |
|---|---|---|---|
| D1 | "all the variables": two meanings in treadcore | Asked; Paul chose both in one queue | Choosing one |
| D2 | orrery is the fleet's console; treadcore knows what training is | The engine in orrery (generic: jobs, lanes, keeper, stops); the planner in treadcore | One script in either repository |
| D3 | make converge already rotates through the weights | The planner calls suggest.py on the queue's own ledger | A new search |
| D4 | Saskatchewan's run is at another commit with other defaults | The queue keeps its own ledger; merging with saskatchewan's waits for one commit | Merging ledgers built against different defaults |
| D5 | A census is one machine's | A trial is one census on one host; a judge reads the candidate and the best on one binary | Pairing across machines |
| D6 | "stop now" must not leave jobs running remotely | A job runs in its own process group with its pid on the host; stop-now kills the group | Killing the local ssh only |
| D7 | Buttons that change the fleet | Drawn only for an operator with --queue-ctl (orrery's rule: what must not be reachable must not exist) | Buttons for every viewer |
| D8 | "lets see it work" | A sample queue with small censuses on seeds no real run uses | The real settings first: an hour before anything showed |
| D9 | The sample's 57 instant failures | The nproc fix, and a breaker: five failures in a row stop the adding | Retrying failed jobs |
III. Methods
LR 110 A and C: build/queue/sample (config.json, plan.json as in orrery's docs/fleet-train.md), fleet-queue.py ... ensure, orrery --gui --operator ... --queue-ctl ....
IV. Results
| builds | four hosts; calber 171 s with the trainer |
| net round 1 | candidate less bc2 +0.64 +/- 0.40 over 4 seeds: the best stays |
| weight trials, 5 minutes | 70, none failed after the fixes; one past two errors |
| faults | nproc on the Mac (57 failed at once); no breaker. Both fixed |
V. Discussion
The fleet can now be left to train treadcore. It keeps itself fed, keeps itself running, and stops when told, with its results kept.
The sample's numbers say only that it works. Four seeds of 2 minutes cannot tell a weight's effect from luck. The real settings (55 seeds of 900 s, and 400 held out) are what can.
The reward-term placeholders, and merging this queue's ledger with saskatchewan's run, are next.