A new class of attacks has been recently discovered that relies on the way the 2.6 Linux Kernel (2.4 does not do this) executes a setuid() call. If I hit my nproc limit (ulimit -u), then if root tried to setuid(myuid), the setuid() fails, returning a -1. There are a number of programs that aren't checking the return value of setuid() calls, which causes the program in question to continue execution as root, even though it thinks it's running as a normal user.
I've been spending the past few days to verify that these issues
are not exploitable on RHEL or Fedora Core. My analysis says they're not. This was important since an X update is non trivial for RHEL and FC4 (it's one big giant package). On FC5 an update wouldn't be so bad since it's just one small self contained package. I've learned a great deal about user limits and how they work in Linux.
User limits are not enforced by default. Children processes inherit them from the parent processes, which in most instances is fine. My initial testing was a simple suid root program that tried to setuid(), which never failed. I expected this to fail since I had a test user with a nproc hard limit of 2 processes, and the test user was logged in twice already. I needed to initialize pam in order to properly set the user limits, since a suid root program will inherit the limits of root, not of the user who you are switching to (which does make sense). I should also note that simply calling pam is not enough, you also have to have set pam_limits in the process' pam configuration file.
The most surprising thing I learned was that the 2.6 kernel sets an arbitrary hard user limit for each user. This bit of code exists in the kernel:
That limit gets set in init, then is inherited throughout the system. This means that every user has an nproc limit, which also means that the setuid() attack will work on every 2.6 kernel system, even if user limits are not enabled.
I wouldn't call it a new class of attacks. People have known about this before. there used to be some codepath in the linux kernel which also allowed this (this was a couple of years so), it has since been changed, this got mentioned in david wheelers secure programming for linux and unix HOWTO.
The whole thing basicly comes down to checking returnvalues in general. If posix says something can fail, then assume it can, no matter what a specific unix kernel does. another nice example here is the close() systemcall. posix says it can be interrupted by a signal. Most unices don't implement this, but they could! Incase some unix does this, it could basicly lead to fd leaks. Which could in turn lead to DoS'es (filing up the fd table for a process or even the whole os), fd leaks in suids (which could very well turn into a local root bug, for example, what if some suid that opens /etc/passwd read write leaks that fd to a process it spawns), or potentially fd_set overflows.