Today I spent the day debugging why a database server wouldn’t boot and, eventually, removing the Redhat kernel from all of our CentOS 7 virtual machines.
Why would I go and do a damn fool thing like that? Well, it turns out that that the RHEL/CentOS 7.4 kernel doesn’t work with XenPV virtualization, which we use in-house and some of our cloud providers use. Not just small ones either – Amazon’s M3 and C3 instances are likely affected.
Usually upgrades are OK to roll out, and in this case it wasn’t noticed until the upgrade also seemed to cause a problem with libm (from glibc) so the system was rebooted, and we were presented with:
[ 1.970630] ------------[ cut here ]------------
[ 1.970651] WARNING: CPU: 2 PID: 225 at mm/vmalloc.c:131 vmap_page_range_noflush+0x2c1/0x350
[ 1.970660] Modules linked in:
[ 1.970668] CPU: 2 PID: 225 Comm: systemd-udevd Not tainted 3.10.0-693.1.1.el7.x86_64 #1
[ 1.970677] 0000000000000000 000000008cddc75d ffff8803e8587bd8 ffffffff816a3d91
[ 1.970688] ffff8803e8587c18 ffffffff810879c8 00000083811c14e8 ffff8800066eb000
[ 1.970698] 0000000000000001 ffff8803e86d6940 ffffffffc0000000 0000000000000000
[ 1.970708] Call Trace:
[ 1.970725] [<ffffffff816a3d91>] dump_stack+0x19/0x1b
[ 1.970736] [<ffffffff810879c8>] __warn+0xd8/0x100
[ 1.970742] [<ffffffff81087b0d>] warn_slowpath_null+0x1d/0x20
[ 1.970748] [<ffffffff811c0781>] vmap_page_range_noflush+0x2c1/0x350
[ 1.970758] [<ffffffff811c083e>] map_vm_area+0x2e/0x40
[ 1.970765] [<ffffffff811c1590>] __vmalloc_node_range+0x170/0x270Fortunately, Kevin Stange at Steadfast already hit this and managed the bug reports at CentOS and Redhat, so we have an identified bug and a patch now. The root bug was a conflict between the Xen MMU and the Kernel Address Space Layout Randomization, and the problem was actually fixed in kernel-mainline in 2014, but the problem we’re facing now is that Redhat is doing major releases without even testing a boot on a major virtualization platform (but one that is not its own commercial offering).
What’s most troubling is the way this has been handled. This problem isn’t documented in the RHEL 7.4 Release Notes and reportedly the request to fix it has been denied by Redhat management. The Redhat bug is locked so people can’t see what’s going on (the CentOS Bug Tracker does Open Source right) and since CentOS Plus is not enabled by default, people running PV DomU’s are going to hit this, rather than avoid it with automated updates.
Frustratingly enough, we’re still using PV in many places rather than PVHVM because that mode isn’t supported by Redhat kernels either, so it’s not like we can “just switch” in performance-sensitive scenarios, like this database server. Recent disclosures about improved security in PVHVM mode has been putting pressure on us to move those services to Debian, and this situation will probably be what pushed us over the edge.
Besides providing a fix for this issue quickly (it’s been out for two days now) in the CentOS Plus repository, the CentOS team also identified the problem via their internal QA process, so there really are people out there who know how to do it right. On the community team. Kudos to both the QA and Engineering teams on the CentOS Project.
Besides the Xen bug, if you do rpm -q --changelog kernel-plus you’ll see there are a few other bugs fixed in kernel-plus that aren’t available in kernel, even on the CR channel. A bug like this one is evidence enough for me that the CentOS team is trustworthy and they are getting bug fixes out to community users much more rapidly than commercial users get. Did I mention they are really doing Open Source right? For this reason I am going to be using the kernel-plus kernel preferentially over the stock kernel in the future. Timely bug fixes are critical to productivity.
So, to actually roll out this fix (manually, you might want to push this with your devops product), as root:
- Edit
/etc/yum.repos.d/CentOS-Base.repoand enable it for kernel-plus. It should look like this:
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=1
includepkgs=kernel-plus
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7- Edit
/etc/sysconfig/kerneland change the default kernel name:
# DEFAULTKERNEL specifies the default kernel package type
DEFAULTKERNEL=kernel-plusyum install kernel-plus
[in theory not needed] `grub2-mkconfig -o /etc/grub2.cfg`
reboot. Select the kernel-plus kernel in grub if it’s not the default.
Verify kernel-plus is running with uname -r.
yum remove kernel to prevent further Redhat kernel updates from coming in (unless Redhat changes its mind on this an update could break your system again).
yum update if you haven’t already to finish the 7.4 upgrade.
reboot to verify everything is 5×5.
[optional] start figuring out your OS/virtualization strategy for the coming months.