A glimpse into what a kernel engineer debugs for enterprise customers.

A bank is running a "security" solution that installs kprobes to intercept, among other things, calls to do_execveat_common(), and monitors all the arguments that could have been passed to execveat(). As do_execveat_common() can be triggered not only by userspace, but also by call_usermodehelper_exec(), a kprobe crafted with poor assumptions may result in an erroneous double dereference of what it thinks points to argv**, causing a General Protection Fault.

The kernel is not dumb however. If a GPF is triggered by a kprobe, it is handled gracefully, and nothing happens, and kprobe just returns a safe value. For a GPF to be triggered however, the CPU has to really try to read the wrong memory address first. The address is pretty random each time, meaning it can point to memory regions that are not mapped by kernel, but have some special meaning for a platform.

Enter the platform. It is configured by the hardware vendor in such a way that if an unaligned access to an MMIO region happens, an MCE is generated. And it is not some MCE for a correctable error, but an MCE indicating process context corruption, in other words, it's fatal. So, once it happens, the system dies with a kernel panic.

And this is exactly what the customer experienced. A socket() syscall caused modprobe to be invoked via call_usermodehelper_exec() → do_execveat_common() chain to load the ipv6 module. This triggered a kprobe that dereferenced wrong memory pointer twice provoking a GPF. The kernel began to gracefully handle the GPF, but the platform saw that the second dereference resulted in accessing the MMIO region, and this was an unaligned access, hence the platform threw MCE. And the system died.

It was fun to investigate this and to explain to the customer that three legitimate things in their system being hit together can trigger a crash.

And of course we joked we should have moved the whole case to the networking team, because it's always IPv6.