building nvidia drivers with clang/lto
on this page
a non-lto clang kernel builds the nvidia dkms modules without complaint. turn on thinlto and the same driver fails to build, in two independent ways. both are fixable without patching nvidia’s code or editing packaged files.
the setup
- desktop workstation: intel i7-12700K (alder lake), asus z690 board, ubuntu 26.04.1 lts
- kernel: mainline 7.3-rc7, built with autokernel using clang 21.1.8 and thinlto (
autokernel build ... --compiler llvm --lto thin, i.e.make LLVM=1withCONFIG_LTO_CLANG_THIN=y) - gpus: rtx 5070 Ti (blackwell) and rtx 4070 Ti SUPER (ada)
- driver: ubuntu’s
nvidia-dkms-595-open(595.99.02, the nvidia open kernel modules)
failure 1: gnu ld can’t read lto bitcode
symptom
the build log at /var/lib/dkms/nvidia/595.99.02/build/make.log stops at a partial link:
/usr/bin/ld.bfd -r -o nvidia-modeset/nv-modeset-interface.o ...
nvidia/nv-platform.o: file not recognized: file format not recognizedthe same error hits both nvidia/nv-interface.o and nvidia-modeset/nv-modeset-interface.o.
cause
with thinlto, every object kbuild compiles is llvm bitcode, not elf. nvidia’s kbuild does partial links ($(LD) -r), and gnu ld cannot read bitcode.
dkms 3.2.2 does detect a clang-built kernel and appends LLVM=1, which would select ld.lld. but ubuntu’s nvidia-dkms packaging hard-codes LD=/usr/bin/ld.bfd on the MAKE[0] line of /usr/src/nvidia-595.99.02/dkms.conf (next to CONFIG_X86_KERNEL_IBT=), and that explicit LD= wins.
fix
don’t edit the packaged dkms.conf; the next driver upgrade overwrites it. dkms sources /etc/dkms/<module>.conf after the package’s dkms.conf (see read_conf() in /usr/sbin/dkms: it sources dkms.conf, then /etc/dkms/$module.conf, then /etc/dkms/$module-$version.conf, …). that file sees the already-set MAKE array and has $kernel_config available, so you can write a version-independent override that only touches lld-linked kernels:
# /etc/dkms/nvidia.conf
# Ubuntu nvidia-dkms hard-codes LD=/usr/bin/ld.bfd in its dkms.conf. Kernels
# built with clang LTO emit LLVM bitcode objects that GNU ld cannot read.
# For kernels linked with LLD, use ld.lld; everything else is unchanged.
if grep -q "^CONFIG_LD_IS_LLD=y" "$kernel_config" 2>/dev/null; then
MAKE[0]="${MAKE[0]//LD=\/usr\/bin\/ld.bfd/LD=ld.lld}"
fiverification
rebuild (sudo dkms install nvidia/595.99.02 -k <kernel>) and check make.log: the make command now shows LD=ld.lld and the partial links succeed. stock gcc kernels don’t set CONFIG_LD_IS_LLD, so they keep nvidia’s exact command.
failure 2: objtool rejects nvidia’s prebuilt blobs
symptom
with the linker fixed, the build gets further and then fails in objtool:
nvidia-modeset.o: error: objtool: nvstatusToString+0x3a: 'naked' return found in MITIGATION_RETHUNK buildthere are thousands of these, plus call without frame pointer save/setup warnings in nvidia-modeset.
cause
in a non-lto build, objtool runs once per object file. nvidia’s precompiled objects (nvidia/nv-kernel.o_binary, nvidia-modeset/nv-modeset-kernel.o_binary) are linked in without ever being objtool-checked.
under lto, objtool runs once on each whole linked module (objtool ... --link --module nvidia-modeset.o), so it now inspects the blobs too. the blobs aren’t built with return thunks or the kernel’s frame-pointer conventions, and objtool says so.
running objtool by hand on relinked modules gave:
| module | “naked return” | “call without frame pointer” |
|---|---|---|
| nvidia.o | ~11,141 | — |
| nvidia-modeset.o | ~2,712 | 249 |
the fully open-source modules (nvidia-uvm, nvidia-drm, nvidia-peermem) were clean.
these warnings are only fatal because the kernel config had CONFIG_OBJTOOL_WERROR=y, which passes --werror to objtool. that is mainline’s strict developer default, and a trimmed config inherits it. ubuntu’s stock kernel config has # CONFIG_OBJTOOL_WERROR is not set.
running the same objtool command without --werror exits 0 on both modules and still emits the annotation sections the kernel uses at runtime (.return_sites, .retpoline_sites, .call_sites).
fix
build the kernel with objtool warnings non-fatal, matching stock ubuntu:
# CONFIG_OBJTOOL_WERROR is not setautokernel now disables it automatically for lto builds: whenever --lto is used, it forces # CONFIG_OBJTOOL_WERROR is not set in the generated config.
i also considered disabling MITIGATION_RETHUNK instead. it doesn’t fix the frame-pointer warnings and it changes the kernel’s mitigations, so OBJTOOL_WERROR=n is the smaller, stock-matching change.
why this is safe
- stock non-lto kernels never objtool-check these blobs at all. turning
--werroroff under lto puts you back where every stock kernel already is; the warnings are reported, not hidden. - objtool still writes
.return_sites,.retpoline_sites, and.call_sitesfor the module, so runtime patching of the open-source parts is unchanged. - on this cpu, return thunks aren’t used at runtime anyway. the kernel reports
retbleed: Not affected,indirect_target_selection: Not affected, andspec_rstack_overflow: Not affected, and spectre v2 is handled by enhanced ibrs. check your own cpu before generalizing that part:
grep -r . /sys/devices/system/cpu/vulnerabilities/verification
after both fixes:
dkms status
# nvidia/595.99.02, <lto kernel>, x86_64: installed
lsmod | grep -E '^nvidia'
# nvidia, nvidia_modeset, nvidia_drm, nvidia_uvm all loadedthe make.log still contains ~14,102 objtool warnings. those are expected and non-fatal.
validation on hardware
both gpus worked on the thinlto kernel:
- nvenc hevc encode worked on each card
- pytorch 2.14.1+cu130 bf16 matmul: 96.5 tflops (5070 Ti) and 91.9 tflops (4070 Ti SUPER), identical to the stock ubuntu 7.0 kernel within noise
- a 3-minute concurrent dual-gpu stress (compute plus bidirectional pinned copies) finished with 0 integrity errors and 0 pcie aer errors
related: iommu mode and gpu transfer bandwidth
on this intel box, the default iommu mode was translated (iommu: Default domain type: Translated, dma-fq). the 5070 Ti, on a pcie 5.0 x8 link, measured only 17.1 GB/s host→device and 15.7 GB/s device→host with pinned memory (cuda-event timed, 1 GiB transfers). MaxPayload/MaxReadReq were already 256/512 and aer was clean, so the link wasn’t the problem.
booting with iommu=pt raised that to 27.7 / 28.6 GB/s (+62% / +82%, ~88–91% of gen5 x8 theoretical). compute throughput didn’t change.
the trade-off: in passthrough mode, host devices dma without iommu isolation (vfio passthrough to vms is still isolated). amd’s rocm docs recommend iommu=pt for gpus for the same reason.
one more multi-gpu note: set CUDA_DEVICE_ORDER=PCI_BUS_ID. cuda’s default FASTEST_FIRST ordering made torch’s cuda:0 a different card than nvidia-smi -i 0.
references
i found no upstream or nvidia fix for either failure. related reports: