Held Back on Purpose: the Versions a GTX 1080 Ti Can Run CUDA On, and Nix as a Way to Pin Them

Abstract

A GTX 1080 Ti is a Pascal card (compute capability 6.1), and NVIDIA is retiring Pascal. Almost every version on the CUDA node calber was chosen older than the newest, because the newest would not run CUDA on this card: Ubuntu 24.04 rather than 26.04, driver series 580 rather than 590, CUDA 12.9 rather than 13. This report records each held-back version, the evidence that forced it, how it is held so an update cannot undo it, and what will end it. It then considers Nix (a flake, or nix shell) as a way to pin the build dependencies (the Rust toolchain, the CUDA toolkit, the libraries a GPU program links) per project and per commit, leaving the kernel driver to the host, which is where it has to stay.

I. Objective

1. Say, for each layer from the kernel to the training library, which version calber runs and why not the newer one.
2. Say how each is held, and what to watch.
3. Weigh Nix flakes and nix shell for the build dependencies.

II. Why Pascal forces old versions

III. The versions, layer by layer

ubuntu-drivers recommends nvidia-driver-470 for the card. That is an older series and was not taken: 580 is newer, still supports Pascal, and is what CUDA 12.9 needs.

IV. What ends each hold

• Driver 580: security updates for a retired series stop at some point NVIDIA announces; after that the card is a security question, not a CUDA one. Watch NVIDIA's driver support notes.
• Ubuntu 24.04: standard support to April 2029.
• CUDA 12.9: nothing ends it on this card; it ends the software: a library that requires CUDA 13 (a newer Burn, PyTorch build or cuDNN) will not run here. Each such library is a version to hold too.
• The card itself: a Turing or newer card (compute 7.5 and up) lifts every hold in this report.

V. Burn on Pascal: the probe

Burn's CUDA backend does not ship precompiled kernels; CubeCL generates them at run time and compiles them with NVIDIA's runtime compiler (NVRTC), for the card in front of it. Whether that path still accepts sm_61 is not something the documentation settles, so it is measured: a Rust program on calber (~/code/burnprobe), Burn 0.22.1 with cuda and autodiff:

1. a 256x256 matrix product on the GPU, compared with the same product done on the CPU (ok if the largest difference is under 1e-3);
2. 200 Adam steps on a net of the learned tank's shape (143 -> 64 -> 64 -> 8) fitting a fixed random target (learning if the loss falls by more than a fifth).

Result, on calber, 28 October 2026 (tools/burn-probe, with its Cargo.lock):

matmul 256x256 on the GPU: largest difference from the CPU 5.96e-6 -> ok
step   0: loss 1.0365
step  50: loss 0.7537
step 100: loss 0.3458
step 150: loss 0.1229
200 Adam steps in 4.02 s: loss 1.0365 -> 0.0446 -> learning

Burn 0.22.1 and CubeCL 0.11.3 run on sm_61. The arithmetic agrees with the CPU to 6e-6, and a net of the learned tank's shape cut its loss 23-fold in 200 steps on the card; the 4.02 s includes CubeCL compiling its kernels for the card the first time.

Two things learned on the way:

• Burn 0.22 changed its API. Tensors no longer carry their backend as a type parameter (Tensor<2>, not Tensor<B, 2>); the device is chosen at run time (Device::cuda(0).autodiff()), and Linear and the optimizer are not generic. The probe was first written for the older API and did not compile; it was rewritten from 0.22's own README and source.
• After the program ends, a CubeCL worker thread panics eight times with CUDA_ERROR_DEINITIALIZED: it is still running while the CUDA driver shuts down. The results and the exit status (0) are not affected; it is noise at exit, to be quieted in the trainer (by synchronising and dropping the device's work before main returns).

Had it failed, the alternatives were weaker than expected: Burn 0.22.0 deprecated its LibTorch backend ("will be removed in a future release"), leaving an older Burn and CubeCL pinned to the last versions that worked, or a newer card.

VI. Recommendations

1. Turn off the release upgrade prompt on calber (Prompt=never in /etc/update-manager/release-upgrades), so 26.04 is not taken by accident; tools/cuda-node.sh should do it for a Pascal card.
2. Never install the unversioned cuda or cuda-toolkit metapackages on it: they would pull CUDA 13.
3. Record the version of everything a GPU program links (Burn, CubeCL, the CUDA libraries) beside its results, as the census records its binary.

VII. Nix: pinning the build dependencies

What it would pin, and what it cannot

Flake or nix shell

Why it fits this project

• It is the census's own rule, for toolchains. treadcore freezes the census binary because a different build plays different matches; a flake.lock freezes the toolchain the same way, and the commit that made a result names it.
• It holds old versions after the world moves on. nixpkgs will drop CUDA 12 in time; a locked flake keeps the commit that had it, and that is exactly the Pascal problem.
• It does not touch apt. The driver stays Ubuntu's (held as in III); the build world lives in /nix/store beside it, and can be removed.
• On Copal too, for build dependencies. Nix runs on Alpine (musl) and can provide glibc toolchains there; not CUDA, since a Copal node has no NVIDIA driver.

Costs

• Nix's store is large, and a first nix develop with CUDA downloads gigabytes.
• Unfree CUDA packages are not in the public binary cache's free set, so they are fetched from NVIDIA's redistributables; the attribute names and which CUDA versions a nixpkgs commit carries must be checked at the pinned commit, not assumed.
• Another tool to learn and keep; and the driver half stays outside it, which is where most of the CUDA trouble lives.

Proposed experiment

A flake.nix for treadcore's GPU crate: Rust (pinned), cudaPackages_12_x with nvcc and nvrtc, pkg-config; the host's driver libraries passed in; nix develop on calber builds the same Burn program as the probe, and the probe's result must match. If it does, the GPU work is built that way from then on.

VIII. Open items

1. Done: the Burn probe (V).
2. The release upgrade prompt off on calber, and in cuda-node.sh.
3. The Nix experiment (VII).
4. The CubeCL exit noise, in the trainer.