Discussion in this thread frequently reflected an unwarranted, underlying assumption. Namely, that linux/unix experts are intalling the OS, know how the system will be used, and act immediately after installation to reset default installation resource limits to appropriate levels. It is obvious to me from thread discussion that that assumption is invalid. Rather the discussion suggests many/some respondents had the author's (and my) grasp of the resource limit issue. It is past time for the linux enthusiats touting linux as more secure than windows to get acquainted with the result of parsing the word assume into ass-u-me!
While approach of applying the sane limits out-of-the-box is bettern than the current approach of not having any limits at all, it is still not the right approach.
The correct approach would be to have kernel that is able to handle abuses correctly. Basically, what happens during fork bombing or any other kind of attack on the resources, is that one process is able to block all other processes on the system from gaining access to one of systems limited resources (CPU time, physical memory, and so on). If you took operating system classes at university, this was called starvation of resources. An ideal operating system should never allow for process to starve. No matter how high the load on the system is, process should get its share of resources.
Now, for the operating system to handle this correctly, it would need to keep some statistics on multiple levels of what is going on the system for each process and group of processes (at least grouped by owner and again by common parrent, and possibly by other criteria) for each of the system's limited physical resources. That way, OS can penalize the abusive process, and protect other processes from starvation. In laboratory, you'll see performance degradation (lower benchmark results), of course. However, in real world (you know, the place where we run production servers), the performance degradation will be minimal, and the system will run way more stable and smoother. Actually, the system will be able to run multiple services without danger of one service affecting other services. I find this way more important to have in real world than meaningless benchmark numbers from lab testings.
An example from real world where I regulary see Linux boxes going down is not fork bombing. It is "swap bombing". You just make a Linux box swap a lot, and it becomes dead (to the point where your SSH sessions will get disconnected on network timeouts). I have one box in my basement where simply typing "yum update" will get it down. The very same issue as with fork bombing. Linux does not protect user space processes from each other. And unlike fork bombing, setting ulimits is not going to save you here. Linux ignores ulimit for maximum number of pages a single process can have in physical memory (unless it is mmaped region with specific value of madvise applied to it, as per ulimit man page).
Now, what should (ideally) kernel do in this case? It should detect that there is a single process attempting to monopolize memory for itself. It should dynamically limit the number of physical memory pages that process can have, and lower that process priority for accessing swap. The affected process (the one that went on memory rampage) will get slowed down by this. How much? Not much, it was going to crawl anyhow because it will spend most of its time waiting for slow disk. However, all other processes on the system will be spared from really not needed swapping. The similar protection can be applied to ther resources, such as CPU time. Small penalty for resource intensive process is (IMO) acceptable price to pay to get rock stable system.
The "yum update" problem that crashes the system was on an old MMX box with 96 megs of RAM. Before anybody says "don't run yum on such an old box with so 'little' RAM", I recently made a small programing mistake that brought down P4 box with 256 megs of RAM basically the same way "yum update" kills that old MMX box. Basically 6 lines of code:
open() - create an empty file pwrite() - write a single byte at 1 gig offset mmap() - map whole file (1 gig) into memory as shared madvise() - lie to kernel we'll access the file "randomly" memset() - entire 1 gig region munmap() & close()
So, nothing fancy. The usuall sequence of calls that you can find in so many programs (well, apart from calling memset on mmaped region right away). I think I told kernel access will be random in madvise (might be some other hint, don't remember anymore what was exactly my stupid mistake at this point). This is different than mallocing entire virtual memory kind of attack. This does not consume any swap space, all other programs will have more than enough virtual memory to continue running (but Linux is going to allow you to steal all physical memory, and you are going . Theoretically, I'm not attacking any system resources here. Practically, I did. And I did it so efficiently, that all I could do is unplug the system from the power, and reboot.
The system is going to die somewhere at memset part. Unless your physical memory is in gigabytes range, so that whole region (or at least most of it) can be fitted into the memory. In that case, either start several of those, or if you are running 64-bit processor, simply change steps 2-4 to use 1 terabyte region. Most systems don't have anywhere near terabyte of memory, for sure. The system will recover from this. Eventually. But while we are in memset, it will be almost frozen for anything else. So frozen that even if sysadmin was logged in on the console, he can't do anything to stop this. He can just watch disk light constantly on as system swaps as crazy. Stupid kernel...
Put this in loop, insert fork() somewhere, do I really need to draw everything? And all of this possible becase Linux kernel does not care what is going on the system (so that is can be the "fastest" system when benchmarked)... The fastest to go down, if you ask me...
-- Aleksandar Milivojevic <amilivojevic@xxxxxx> Pollard Banknote Limited Systems Administrator 1499 Buffalo Place Tel: (204) 474-2323 ext 276 Winnipeg, MB R3T 1L7
