August 18, 2026
andy
A couple of weekends back, I upgraded my home sever. It failed to restart after running apt dist-upgrade
The only update that was performed was to the kernel, it went from 6.12.88+deb13-amd64 to 6.12.100+deb13-amd64. I had previously performed an apt-get upgrade, and rebooted the machine, so I was pretty sure that this was to blame. This blog entry (is a late) attempt to document how I went about finding a fix for this issue so that next time I don’t need as much hand holding as I did this time around :-)
(1)
Having my machine not boot following an upgrade is pretty rare, but has happened before. Usually it is because I have done something wrong so as always confirming I haven’t broken something by accident is always my first step…
I plugged in a keyboard an monitor to the machine and watched it boot. Being a server this takes a long time (I guess because at this stage of system initialisation we want to test things sequentially)
Watching the system boot I see the usual BIOS/UEFI stages for this machine, followed by the grub menu and the the local screen showed:
Loading Linux 6.12.100+deb13-amd64 ...
Loading initial ramdisk ...Nothing else. That was it. OK that looks like I have a broken system all right, and at very early stage of the boot process process.
(2)
Breaking into the grub menu and removing the quiet option yields a little more information (but not much):
Loading Linux 6.12.100+deb13-amd64 …
Loading initial ramdisk ...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID d
evice path
EFI stub: Measured initrd data into PCR 9and nothing else.
(3) Initial debugging
- Confirmed that I could still boot the machine with the old kernel
6.12.88+deb13-amd64(During boot select Advanced options from the grub menu followed by the kernel image wanted)- Yes – the system starts happily with the previous kernel
- Checked that /boot had enough space
- Yes – plenty of space
- Is anyone else reporting this problem?
- Nothing jumps out on Debian’s bug tracker
- Actually not mush referenced for my search “
EFI stub: Measured initrd data into PCR 9 apart” other than the usual rantings to “turn off secure boot” (on this server that currently isn’t turned on – bad me)
(4) Triage
Start looking for where the fault first occurred. At this point I needed help, and given that Sledge was visiting I asked if he would sanity check what I was doing. His initial thoughts were that that /boot had run out of space, but replaying my step (3) with him acting as a ‘rubber duck’ showed that this was something other than PBKAC
Sledge had a quick look, then informed me that between kernel images 6.12.88+deb13 and 6.12.100+deb13 Debian stable has only had shipped .90 .94 .95 and .96 kernels. We could easily try them all:
wgeteach kernel package then install (dpkg -i) followed by anupdate-grub, checking that there was sufficient space on disks especially my small/boot partition)- I started with image
6.12.95+deb13and this worked 6.12.96+deb13yielded the same lock up on boot as6.12.100+deb13
OK I now have the first kernel image that doesn’t boot on my system, time to raise a bug…
Up until now I have been walking to my garage where the server is located and standing in front of a rack
with a monitor and keyboard plugged into the machine. However this machine supports IPMI so I spent a little time getting that up and running so that I can continue from the relative comfort of my desk (with lights, a chair and not needing to hold the keyboard with one hand)
Great I can now grab screen shots from the confort of my desk (unfortunatly they are only screen shots not text files, but at least we can seen the early stage of boot, Post, grub menu and then initramfs before system log happens)
(5) Collating information for the initial bug report
Sledge had mentioned my problem in irc/#debain-kernal where iam_tj suggested that we try appending
‘debug earlycon=efifb’ to the kernal command line. This yielded 15 seconds worth of messages before the system locked up the last few messages being (vmlinuz-6.12.96+deb13-amd64):
[ 14.663477] RCU Tasks: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.750474] RCU Tasks Rude: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.838024] RCU Tasks Trace: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.929752] NR_IRQS: 524544, nr_irqs: 584, preallocated irqs: 16
[ 15.016814] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[ 15.104011] Console: colour dummy device 80×25
[ 15.191236] printk: legacy console [tty0] enabled
[ 15.278249] printk: legacy bootconsole [efifb0] disabled
Booting the working kernel with the same kernel options yields the SAME messages with slightly differing times, but then continues to login prompt:
[ 14.697466] RCU Tasks: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.784936] RCU Tasks Rude: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.872067] RCU Tasks Trace: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.964000] NR_IRQS: 524544, nr_irqs: 584, preallocated irqs: 16
[ 15.051482] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[ 15.226079] printk: legacy console [tty0] enabled
[ 15.313751] printk: legacy bootconsole [efifb0] disabled
[ 15.400831] ACPI: Core revision 20240827
[ 15.401415] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[ 15.401464] APIC: Switch to symmetric I/O mode setup
... and so on
iam_tj also suggested adding keep_bootcon – with ‘debug earlycon=efifb keep_bootcon’ on vmlinuz-6.12.96+deb13-amd64:
We get a LOT further – and we see a crash / trace-back:
[ 34.285342] BUG: kernel NULL pointer dereference, address: 0000000000000000I raised bug #1143721 and followed it up with screen captures of the boot sequence (captured from the IPMI client) and files containing the output of dmidecode, lscpu and lspci to kive the kernel team as much information as possible:
[6.12.96+deb13-amd64 debug earlycon=efifb keep_bootcon.tar.gz (application/gzip, attachment)]
[dmidecode.txt (text/plain, attachment)]
[lscpu.txt (text/plain, attachment)]
[lspci.txt (text/plain, attachment)]
(6) Tracking down the bug Git Bisect
The problem with this type of bug is that it is hardware (class) specific, whilst the kernel doesn’t boot on my system, it clearly has worked on machines used by the kernel team, the Debian test and build infrastructure, (otherwise this kernel would never have been released) and everyone else who has upgraded to the newer kernel before I did (otherwise we would be drowning in fails to boot bug reports). Carnil’s excellent response to my bug: Message #15 (and help in IRC) provided me with a detailed step by step guide in how to track down the individual git commit that fails on my system. I had already (with Sledge’s suggestion) made a clone of the stable branch, but was struggling to follow the steps in the Debian Linux Kernel Handbook to re-build a duplicate kernel because I didn’t understand how to obtain the same configuration that Debian used to build the kernel; Carnil’s email provided me the missing steps (Highlighted).
git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git cd linux-stable git checkout v6.12.95 cp /boot/config-$(uname -r) .config yes '' | make localmodconfig make savedefconfig mv defconfig arch/x86/configs/my_def test 6.12.96 to ensure this is "bad" git checkout v6.12.96 make my_defconfig make -j $(nproc) bindeb-pkg … install the resulting .deb package and confirm it fails to boot and triggers the NULL pointer dereference.
Right I can now start to Bisect the problem:
git bisect start
git bisect good v6.12.95
git bisect bad v6.12.96Rather than use the half step point’s git bisect suggested I was advised in irc to jump straight to the a given commit that from the git log was suspected as the culprit:
git checkout 977855894bca4b87afa50d21e3f3e85a5a0e901f
build and install….
fails…
git bisect bad
git checkout 977855894bca4b87afa50d21e3f3e85a5a0e901f~1 ## ~1 is the commit beforehand
build and install….
fails…
git bisect good
The entire test tree can shown with git bisect log and this was submitted as an email to the bug report, we have found our smoking gun :-)
Finally I would like to thank Carnil, Iam_tj for their time patience and fantastic support in guiding me through finding this regression. Right now kernel bugs are coming in thick and fast with a lot of AI assisted bug hunting, the increased numbers of bugs mean that the kernel team are especially busy. Hopefully our paths will cross and I’ll be able to buy you some beers (or whatever) soon. thank you. Sledge also deserves thanks for putting up with me and pointing me in the right direction (as ever). Lucky for me that he lives nearby so I can provide beers on a regular basis :-)