Free UNIX-derived operating systems have traditionally have a simplistic approach to process synchronization which is unsuited to multiprocessor application. Initial FreeBSD SMP support kept this approach by allowing only one process to run in kernel mode at any time, and also blocked interrupts across multiple processors, causing seriously suboptimal performance of I/O bound systems. This paper describes the work done to remove this bottleneck, paying particular attention to the project management aspects and the particular challenges of a large open source development project.
Improving the FreeBSD SMP implementation
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSAlert.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
*Free* UNIX-derived OSs had this problem. SysV, and all the commercal UNIXs derived from it, was designed for excellent SMP performance.
“*Free* UNIX-derived OSs had this problem. SysV, and all the commercal UNIXs derived from it, was designed for excellent SMP performance.”
Wrong… SysV systems/commercial UNIX’s where not designed for excellent SMP performance anymore than they were designed for 64 bit computing. They just moved on this concept much earlier.
SysV R4.2 had an MP version in the very early 1990s. From the beginning SysV R4 followed a model of organizing the kernel as a series of cooperating threads, and had a fine-grained locking model in place to support kernel preemption. Supposedly, SysV’s suitability for MP was one of the reasons Sun moved from BSD to SysV for Solaris.
Why are they so behind? I can understand linux being behind but FreeBSD? 1984? seems like plenty of time to develop good support. I bought a 1GHZ dual box and promptly installed freeBSD 5.1 on it and compiled a MP kernel will i get better performance from Linux?
Currently, yes.
In the near future, probably not.
Why are they so behind? I can understand linux being behind but FreeBSD? 1984? seems like plenty of time to develop good support. I bought a 1GHZ dual box and promptly installed freeBSD 5.1 on it and compiled a MP kernel will i get better performance from Linux?
Unfortunately FreeBSD has never been able to get the same hype and mindshare which LInux has, also, since there is no “commercial” wing as such, there are no full time programmers dedicated to just working on FreeBSD.
What FreeBSD does need to do is create a commercial wing of FreeBSD; make a distribution based on FreeBSD but easier to install and configure the way how SuSE does it. Create a “control centre” which allows the configuring of a server and workstation via a centralised tool, include documentation, enhance OpenOffice and rebrand it “FreeBSD Office Suite”, and include fonts, clipart and so forth then bundle it with the desktop version.
With a “commercial wing”, there will be a reasonable revenue stream coming in to pay for 4-6 full time programmers who can dedicate themselves to fixing these issues. As I said to FreeBSD developer, I am willing to do all the heavy lifting to get it up and running. It is a shame that I was ignored and here we are with FreeBSD atleast 4 years behind Linux and increasing by the day.
You need to re-compile kernel for SMP support.
<quote>
Unfortunately FreeBSD has never been able to get the same hype and mindshare which LInux has, also, since there is no “commercial” wing as such, there are no full time programmers dedicated to just working on FreeBSD.
</quote>
Wrong! There are full-time BSD developers.
<quote>
With a “commercial wing”, there will be a reasonable revenue stream coming in to pay for 4-6 full time programmers who can dedicate themselves to fixing these issues. As I said to FreeBSD developer, I am willing to do all the heavy lifting to get it up and running. It is a shame that I was ignored and here we are with FreeBSD atleast 4 years behind Linux and increasing by the day.
</quote>
So they can whore themselves out for money like Mandrake is forced to do? No thanks. You were ignored because people who’ve probably thought about it alot longer and harder than you have decided it was a bad idea.
Commercializing FreeBSD would likely result in the same thing that happened to BSD/OS.
“It is a shame that I was ignored and here we are with FreeBSD atleast 4 years behind Linux and increasing by the day. ”
FB is definitelly not 4 behind Linux. You can do with FB everything You can with Linux and even more.
It is fast, it is more stable than Linux, and I use it as my primary machine, my workstation, my VCR and server serving 2 other machines at home … and I never needed Linux.
(note to the Linux camp: of course Linux’s SMP implementation is a remarkable technical accpmplish, but take a few seconds of breather from tooting your favorite operating system’s horn to appreciate what has gone into FreeBSD 5)
In order to comprehend the sheer amount of design an implementation work that went into SMPng, read this document which describes where they were starting from:
http://www.lemis.com/~grog/SMPng/USENIX/paper.ps
SMPng started with the BSD/OS 5 implementation, which began to move away from the Giant lock, allowing kernel space to be entered without acquiring the Giant lock. FreeBSD has since proactively moved to fine grained locking, to the point that most of the kernel proper no longer needs to touch the Giant lock. The Giant lock remains primarily for a number of device drivers. You can track the effort to remove Giant from these drivers here:
http://www.freebsd.org/projects/busdma/
Today, FreeBSD 5 has the best SMP implementation of any BSD derived operating system (although this is arguable if you were to count the Mach-derived Tru64 as being a BSD derived operating system) It has greatly transcended its roots in the BSD/OS 5 implementation, and is beginning to show itself as a robust and scalable multiprocessor operating system.
Scheduler entities are already proving to be radically more scalable than the previous userpace threads implementation in libc_r. Many applications can already be compiled with libkse, and given the current development progress we are almost certain to see FreeBSD 5.2-RELEASE sporting KSE threads as the primary systemwide threading implementation.
I would like to congragulate the FreeBSD 5.x team for the remarkable technical accomplishment they have managed to pull off, and hope to see continued improvements in the future.
I haven’t read so much about UNIX SMP and how unix handles processes, threads, and the kernel before. Thanks for posting this.
Its a little sad reading about how slow the development is going in FreeBSD. I’ll probably still be using FreeBSD 4.X for a while. Even if I only have one CPU.
So Coo,
What you are saying is that Apple is not contributing in any way, to the development of FreeBSD in spite of benefiting from the use of FreeBSD?
In the August 2000 issue of the Daemon News ezine, Greg Lehey explained the SMP issue. It’s a nice summary for those who are interested.
If System V was so good for multiprocessor use, then what sort of multiprocessor minicomputers was it running on when it was released?
Not to be offensive, just curious. I’ve never heard of multiprocessor minicomputers that System V was capable of running on in such a way that it could utilize all the processors in the system
I bought a 1GHZ dual box and promptly installed freeBSD 5.1 on it and compiled a MP kernel will i get better performance from Linux?
I believe the answer is, in general, that yes, overall you will get better performance from Linux 2.6 than you will get from FreeBSD 5.x
There are a few areas where my own benchmarks have shown that a GENERIC 5.1-RELEASE kernel outperforms a Linux 2.6-test4 kernel with as few configuration changes as possible from the default configuration to get the kernel to boot on my hardware. The areas where my benchmarks show FreeBSD outperforming Linux are heap/VMM performance and dbench throughput perfomance. Of course, any benchmarking which reflecting Linux in a negative light is enough to stir the Linux trolls from their slumber, at which point they will call me a liar and do everything they can to prove my benchmarks conceptually flawed; however for some reason they refuse to repeat the benchmarks themselves on both FreeBSD and Linux and post their numbers (once one of them actually posted numbers for Linux, but didn’t post numbers for FreeBSD)
So, let me just say this instead of posting my own benchmarks yet again… (they’re rather outdated by now anyway) in my experience a heap-bound application will, in general, perform better on FreeBSD.
Of course, there are several respects in which a direct VMM to VMM comparison shows Linux’s to be superior (such as a recent benchmark of Linux vs. FreeBSD mmap performance) I believe at this point the difference in heap performance between FreeBSD can be traced not to the kernel but rather to glibc. glibc 2.2 saw the heap implementation [for you semantic trolls out there, I’m referring to the implementation of malloc()/calloc()/realloc()/free()] scrapped and reimplemented from scratch, but from my previous tests and in my own personal experience this was not enough to gain a performance advantage over FreeBSD.
The heap implementation in glibc 2.2 was scrapped, and rewritten from scratch (as far as I know) for glibc 2.3
Setting up the network stuff was for me the big nightmare in FreeBSD. Sure I realize FreeBSD is made for servers and from recent looks of longest uptime on NetCraft, FreeBSD and BSD/OS keeps all 50 slots of longest uptimes without competition.
But for me, who isn’t a poweruser and find network configuration very tricky but wanna run BSD on my desktop just can’t get passed that barrier. A control panel would help me greatly.
Can’t someone just make an .iso with X preconfigured and network setupd sort of fixed or so??? I’d love to be able to run Fluxbox and Gnome and that’s pretty much it. Ports seems very simple to handle so that won’t be a problem.
Now someone might say that I should go with Linux… but sorry I’m frankly not interested in Linux. Many distros might work out of the box, but the attitude of the community is the kind of attitude I don’t wanna be around.
Anyone have any ideas how I can get passed the configuration barrier?
But for me, who isn’t a poweruser and find network configuration very tricky but wanna run BSD on my desktop just can’t get passed that barrier. A control panel would help me greatly.
The closest thing to a “control panel” on FreeBSD is probably /stand/sysinstall
To change your network settings, go to Configure > Networking > Interfaces
Or you can simply edit the appropriate ifconfig lines in /etc/rc.conf
from recent looks of longest uptime on NetCraft, FreeBSD and BSD/OS keeps all 50 slots of longest uptimes without competition.
Netcraft is incapable of reporting uptimes past 497 days for operating systems other than FreeBSD and BSD/OS:
http://uptime.netcraft.com/up/accuracy.html#cycle
I would consider the uptimes on those FreeBSD and BSD/OS systems more indicative of poor administration than OS stability. Those systems are certain to be littered with dozens of known security vulnerabilities and other issues as the administrators clearly are not patching them.
A question (not meant to offend you though) to you – what makes u think that those long-running machines are running with vulnerabilities and without any patches applied? Most of the vulnerabilities are from applications/daemons which can be patched without rebooting the system.
this is not entirely true. Read freebsd hackers forum. Even A.Cox suggested that there is difference between optimizations for some of the benchmarks and real world apps. Still there is a lot to improve in terms of mmap.
when I look at all the auditing and effort put into OpenBSD I can’t help but wonder why no-one has built a smoothwall or clarkconnect “webmin for dummies” interface (apart from the fact that wouldn’t have been audited in the same way). would not thousands of homes protected by OpenBSD not be a “good thing”?
It still seems to be “In progress” in SMPng site…
http://www.freebsd.org/smp
John Baldwin has been working on it a long while ago, there was some code which didn’t work for some reason. But then all this seems to have disappeared. I don’t get to hear about this anymore. I wish it is completed soon.
SMPng might be great, but on my 5.1 system, I can’t use anything that does heavy network usage without my system locking up after a few minutes. I looked around, and it looks like all the fun SMP stuff isn’t quite done for things like, oh, networking.
So, I’m back to the bsd 4 scheduler.
FreeBSD 5.1 worked great on one of my systems, but not so great on the other.
I had serious problems with background fsck when power went out one night during a storm (of course I had to be compiling the kernel at the very moment power went out).
Anyway bgfsck didn’t work and made me piss around doing some forced fsck in single usermode. I recall spending half an hour hitting the enter key trying to repair some obscure lowlevel file system thing. Needless to say this scared me away from 5.1 for a while.
Also I found that bgfsck (when it did work) really slowed the system down — to a very painfully slow speed (stealing all the disk bandwidth I suppose). I recall that maybe the new scheduler is supposed to take this into account?
Has this been fixed yet or forgotten about?
I don’t really understand your question, but check on John Baldwin has posted about his very huge work yesterday in freebsd-arch mailing list.
http://lists.freebsd.org/pipermail/freebsd-arch/2003-October/001421…
Hope, it’s what you want to hear.
So, I’m back to the bsd 4 scheduler.
You can try the today ULE scheduler that Jeff just has committed. It’s a lot better!
Check here: http://lists.freebsd.org/pipermail/freebsd-current/2003-October/012…
It will be nice if you give it a test and report to him the result. The report is always important that can get him to fix the things to improvement.
A question (not meant to offend you though) to you – what makes u think that those long-running machines are running with vulnerabilities and without any patches applied?
Well, looking at http://uptime.netcraft.com/up/today/top.avg.html the first clue is that the top system is running Apache 1.2.4, which is affected by the chunk handling vulnerability:
http://www.cert.org/advisories/CA-2002-17.html
Well, looking at http://uptime.netcraft.com/up/today/top.avg.html the first clue is that the top system is running Apache 1.2.4, which is affected by the chunk handling vulnerability:
http://www.cert.org/advisories/CA-2002-17.html
Well, don’t let the pretty numbers to fool you. They might have or not have patch their Apache 1.2.4. Unless, you just have tested to use the vulnerability on them. ;-P
I don’t really understand your question, but check on John Baldwin has posted about his very huge work yesterday in freebsd-arch mailing list.
Go: http://www.freebsd.org/smp
Search for string “Make the kernel fully preemptive”…
I don’t think the other work is related to fully preemptive kernel.
Its a shame that an OS which was designed from day one for SMP died due to lack of commercial/consumer support, yet these ancient OS’s get hacked and fudged to support SMP, and blossom. The worst part is that all these SMP implementations still work around a global lock to access kernel land (no preemptive kernel), due to all synchronisation nightmares. Only recently have they started working at preemptive kernels, and they’re still light years away from where BeOS was 8 years ago.
Its a masochistic world, I tell’s ya.
The worst part is that all these SMP implementations still work around a global lock to access kernel land (no preemptive kernel), due to all synchronisation nightmares.
The Big Kernel Lock in Linux 2.6 is virtually irrelevant, used by a scant number of legacy subsystems and may be removed before the final release; I’m unable to find any current, pertainent information as to what subsystems it still remains in.
As for FreeBSD, work removing its Giant lock isn’t quite so far along, but it’s used primarily in the driver subsystems, and from everything I’ve seen the presence of this lock does not affect the throughput of these drivers. On the BeOS side, however, throughput is pathetic…
Only recently have they started working at preemptive kernels, and they’re still light years away from where BeOS was 8 years ago.
BeOS had what was perhaps the lowest I/O throughput of any modern operating system. Its scheduler was quite responsive, and it had great systemwide policies for use of threads. However, throw any I/O intensive activity at it and it crumples, due in part to the high number of context switches necessary for the kernel to talk to the userland servers. Network performance would’ve been partially remedied by BONE, but this still leaves pathetic throughput from the disk subsystems.
Keep and mind that Linux and FreeBSD are both used and servers, and would be unacceptable for server use if their kernels had such horrendous throughput bottlenecks.
If you make smp to good, SCO will say it is theirs…. How could one of the BSDs do this on their own, they must have ripped it off.
I can see it now ;-(
Its a shame that an OS which was designed from day one for SMP died due to lack of commercial/consumer support, yet these ancient OS’s get hacked and fudged to support SMP, and blossom. The worst part is that all these SMP implementations still work around a global lock to access kernel land (no preemptive kernel), due to all synchronisation nightmares. Only recently have they started working at preemptive kernels, and they’re still light years away from where BeOS was 8 years ago.
Wake up. BeOS’s SMP was largely terrible from a performance standpoint. I might also note that BeOS has a global lock around the semaphores, lacks CPU affinity support, and a number of other things which contribute to quite lackluster SMP performance in practice. What Be’s marketing said and what the kernel was in fact capable of are two very different animals. Its perceived speed/responsiveness was due to some very clever tricks on the part of the scheduler and app_server, but it was by no means a fast SMP system, just responsive.
> Its[BeOS] perceived speed/responsiveness was due to some
> very clever tricks on the part of the scheduler and
> app_server, but it was by no means a fast SMP system, just
> responsive.
You’re right of course, but IMHO responsiveness is more important than throughput for desktop usage!
The responsiveness of Linux I’ve used (straight 2.4 or 2.4+preemptive patches) is laughable compared to BeOS.
It is hard to say if it is caused by the kernel (scheduler), the GUI, or the apps though, maybe all the parts could be improved..