A Dead Clock Battery and an Expired Certificate: Why jaguaribe Left the Fleet

Abstract

jaguaribe, a Supermicro X8DTL with two Xeon X5670s, did no work for 17 hours and 54 minutes after the fleet's shutdown on 31 October. Paul found a CMOS error at the console and a dead clock battery. The machine's own log shows it booted believing the date was 16 November 2012, and that chronyd stepped the clock forward 440,470,337.786 seconds, about 13.96 years, 26 seconds after boot. The date is now correct, and the system clock and the clock chip agree to the second. It is not fixed for good: the next loss of mains power will reset the chip again until the battery is replaced.

The logs also settled a second question. The SSH resets seen on all three Copal machines are not a network fault. orrery's chat room dialled every machine every 4 seconds with an operator certificate that expired on 30 October. Each refusal cost the Mac's address a penalty in OpenSSH 10.3, and sshd then dropped connections from that address, the keeper's among them. The three Ubuntu machines run OpenSSH 9.6, which has no such penalties, and logged none.

I. Introduction

The fleet's keeper had reported jaguaribe unreachable since 31 October. An earlier reading in this work described it as dropping off the network repeatedly. That was wrong: there were two separate faults, and one logging bug that made them look like one.

II. Method

Everything was read as the ordinary user, with no change to the machine: /var/log/messages, /sys/class/rtc/rtc0, /sys/class/dmi/id, chrony's configuration, file times under /etc and ~/.fleet-queue, /proc/stat, and the queue's events.jsonl. The other machines' logs and ssh -V were read for comparison.

What could not be read. /var/log on Copal is a tmpfs, so every log from before this boot is gone. dmesg and /var/lib/chrony need root. A subnet sweep and a look at the router's leases were refused by this session's permission system and not attempted again.

III. Results

A. The clock

The date the clock fell back to is the BIOS's own build date. That is the usual sign of a chip that lost its contents.

Files under /etc carry two wrong dates: 15 November 2012 at 16:46 PST, and 16 November at 07:03 PST. So there were two boots on the wrong clock, about 14 h 18 min apart.

B. A probable timeline

The inferred times count back from the measured boot, and assume the chip ran steadily. saskatchewan also booted about 16:52, which fits but does not prove it. Why the 17:38 boot never answered the keeper is unknown: its logs died with it. A BIOS waiting for a key at a CMOS error, which is what Paul saw, would explain the hours after it. It does not explain that boot itself.

C. What the wrong clock can break

Between boot and the step, 26 seconds here, sshd was already listening and the clock was 14 years behind. Reasoned from how these checks work, and not tested: a certificate is not yet valid to it, HTTPS fails for git fetch and for packages, and a census stamped then sorts as 2012. If the internet were down at boot all of that would last until it returned. Here it did no harm. Nine files in /etc keep 2012 dates, harmlessly, and the build ran after the step.

D. The SSH resets

The log lines are exact:

error: Refusing certificate ID "user@users-iMac.lan" ... Certificate invalid: expired drop connection #0 from [192.168.8.143]:60164 ... penalty: failed authentication

The certificate lasted to 30 October 19:30. Keeper events of "host unreachable", from 29 October to that time and from then until today:

The windows are about equal in length, though the fleet was stopped for part of each.

E. The false "host back"

A lane waiting on a machine logs "host back" whenever the wait ends, including when a keeper restart ends it. The entries for jaguaribe at 07:12 and 07:16 on 1 November were this: it was off. The health line "every machine answers" at about 07:13 was probably the same. Not yet mended.

IV. Changes Made

1. The date. Nothing was needed by hand: chronyd had corrected it, and the chip with it.
2. tools/orchestrate.py. The chat room and the minute's survey no longer dial on an expired certificate. Measured: with the chat room's guard in at about 07:59, penalty drops stopped on all three machines (last at 07:58:58 and 07:59:07) and refusals fell from one every 4 s to about one a minute, the survey's. The certificate was renewed at 08:00, and there has been no refusal or drop since 08:00:24, nor any keeper "unreachable" since 07:19. The survey's guard went in at 08:01, after the renewal, so it has not been seen working against an expired certificate.
3. Earlier today, in tools/fleet-queue.py: retries on ssh's failure, so a reset cannot lose a finished census.
4. The operator certificate lasts 720 hours (thirty days), not 8, at Paul's asking. It is adjustable: copal fleet login --hours N once, or a number in the fleet's cert-hours file for every login and Orrery's Renew. The price is stated in docs/lockdown.md: a stolen operator key holds the fleet for up to that long.

The queue took jaguaribe back by itself, with no change to any record.

V. What Needs a Person

1. Replace the battery. On this board that is most likely a CR2032; check the holder before buying. Then confirm the date in the BIOS.
2. If the BIOS offers it, stop it halting on errors, so that a machine with no one at it still boots.
3. Decide how long the operator certificate should last. Thirty days by Paul's decision (Section IV). Shorter is safer if the Mac is ever lost: write hours in ~/.copal/fleets/grove/cert-hours, then press Renew.

No ssh-copy-id is needed anywhere: the fleet's user key is accepted on every machine read here.

Until the battery is in, leave the machine's mains power on, since the chip is kept by standby power.

VI. Limitations

• The battery's death is Paul's reading at the console, and the logs agree. It was not measured.
• Other machines' batteries were not tested. Their chips agree with their clocks, but rtcsync makes that so on any running machine. Only a boot after a power loss shows a weak battery. saskatchewan and columbia keep under an hour of log, so an old step would no longer be there to find.
• The penalties are shown to happen and to drop connections. How much of each unreachable count they explain was not separated from the machines' real absences.
• The penalty settings themselves (sshd -T) need root and were not read; the behaviour is in the log.

VII. Future Work

• Persistent logs on Copal: this report's central question could not be answered because they vanish at each boot.
• Mend the false "host back".
• The guard covers the chat room. The minute's survey and button presses still dial as they did.
• A warning before the thirty days run out: the guard stops the penalties, but the chat room then goes quiet.