The next kernel security bug: three fixes beyond patching

This year brought a string of critical security bugs across many components, including the Linux kernel. First Copy Fail, then Dirty Frag, then they started arriving in groups. Can we do better than just patch and wait for the next one that's around the corner?

Read the mitigation recommendations one after another, and the result is a slightly comical pattern: Disable this module. Patch. No, actually also disable this module and patch. Then also disable these 3 modules. Don't forget to patch.

While patching is useful, it does typically only remove the known attack vectors, not the next unknown bug. To address unknown bugs, we traditionally employ a different set of tools: attack surface reduction informed by threat modeling.

After a bit of thinking, I came up with the following three countermeasures: freezing the kernel module set, demoting unnecessary privileged programs, and network egress control. These measures don't replace patching, but they alleviate the pressure under which it must be done. 

Freezing the kernel module set

A common characteristic behind many of these bugs is that they exist in a piece of functionality—a module—that may not be required for typical use, but it is included by the distributor for convenience. The library of these modules is vast and a bug in any of them can lead to the same consequences regardless.

The Linux kernel can automatically load a module when a feature is requested but its module isn't present. Thus, the exploit works this way: an unprivileged program requests an obscure function, which causes the kernel to load the affected module. That module is then exploited, leading to full system compromise.

Instead of disabling one vulnerable module with each bug report, we can reverse the approach: load only what is really required and prevent any additional modules from being loaded on request. That's “hardening”, or attack surface reduction. Luckily, on most systems this can be done using a rather short procedure:

• Sample the current module set using lsmod after all required functionality has been invoked
• Extract the module names and put that list under /etc/modules-load.d/ to be loaded at startup
• Create a one-shot service that runs after the modules are loaded and performs sysctl -w kernel.modules_disabled=1

The last step prevents modules from being loaded or removed until the next reboot, thus locking in the current set. Therefore, the exploit cannot load and reach the buggy code, unless of course, that happens to be one of the modules your host requires.

Demoting unnecessary privileged programs

Exploiting a kernel bug may only be the first step. The attacker still needs to use that access to create a useful privilege escalation in userspace. The Copy Fail bug did this by launching su, a privileged program, while simultaneously changing its loaded code to not perform the traditional password checks and give you a shell directly instead.

Like with the kernel modules, distributions traditionally ship with many such privileged programs. Each one represents an additional path to system compromise. When a privileged (“setuid root”) program is run by a user, the kernel just runs it under the high privilege and then relies on the program to do the right thing. If the checks can be circumvented, or its code overwritten like with copy fail, then its high privilege can be abused for many other things.

A case in point is the ssh-keysign program. Its manual page says “ssh-keysign is disabled by default”, but that only pertains to the functionality it represents, not to the fact that the attack surface is absent by default. So in this case, everybody inherits this privileged helper, even when they don't use it for anything meaningful.

Another aspect of the distribution convenience is that all these privileged programs are typically available to be run by everyone. In reality, the need may be for the actual administrator only, not for the account under which services are run.

So the next surface reduction is to remove the special privilege from the programs that you don't require, and to restrict who can run the remainder. A script illustrates the point:

# Remove all SUID bits from package/application trees
find / -xdev -type f -perm /u+s -execdir chmod -v u-s -- {} \;

# Re-create the permitted privilege transition for the required programs
chown -v root:yourgroup /bin/{whatever,you,require}
chmod -v 4710 /bin/{whatever,you,require}

This straightforward script is the easy part of the process. Identifying which privileged transitions to keep might be harder depending on the complexity of your system.

I also discovered that the package manager tends to restore those changed privileges, so they need to be re-checked after upgrades.

Network egress control

This part is the one that is harder to implement, and it addresses a slightly different set of concerns, but pays off twice: before and after exploitation.

It also pertains to a convenience not of the distribution, but of the environment where the service is run: many environments permit unrestricted connections anywhere—the Internet, other services in the same environment, or even other environments. Unless something is done about that.

Restricting the outbound connections is better done off the device itself: on a network device such as the switch, router, or firewall. If that's not available, it's possible to do it on the Linux device as a stopgap, but a skilled attacker would probably be able to disable it, once they gain a foothold.

Suppose there is a remote code execution bug in a service that the server is running. While it's technically possible to just use that one incoming connection to conduct the attack, most attacks don't do that: they use the RCE to either download more payload or to launch an outgoing command and control connection back to the attacker. This is based on the assumption that the server has unlimited egress. If that's not the case, some of these attack paths become either unavailable or at least harder and more visible.

The other way this pays off is when an attacker actually manages to compromise the server fully. Good egress control significantly narrows the options of how they can continue locally—what other servers or services are available.

Think about authority, not privilege

In security, we're used to thinking about privileges. Even a well-known principle is called “least privilege”. However, this thinking leads to a slightly misaligned set of assumptions and questions. It assumes privileges exist, and asks how an entity can be prevented from advancing to the next rung.

Thinking about authority instead asks a different set of questions: what is currently available to the entity at the current privilege level? Which capabilities are inherited, enabled by default, or present just as a convenience? And ultimately, how does that compare to the function that is requested?

The three legs of the milk stool that I described are authority restrictions, not privilege hardening. We simply remove what was inherited by convenience, and in doing so, we lessen the chances of something going wrong for unknown reasons.

A common pitfall of removing authority is the “putting the worms back into the can” problem: we can inadvertently remove a piece of real functionality that something or someone depends on. I would characterize the three measures as moderately invasive: the restrictions are real, but the baseline required is not overly complex. Iterative testing, observability, and starting with a wider baseline were the components of my safety net.

What I like about this approach operationally is that it does not require abandoning the benefits of a Linux distribution. While it would've been possible for me to create my own distribution with a minimal kernel and set of privileged programs, it would've also brought a substantial maintenance burden. This way, patching, building, and testing stay with the distributor, but the user has a say in what functionality is authorized.