Guest post by The DragonFly BSD operating system is a server oriented project. Though originally forked from FreeBSD, DragonFly BSD is an independent operating system that carries a number of unique features, foremost among them is the HammerFS file system. DragonFly BSD 4.0 was released on November 25th and offers several new features.
Version 4 of DragonFly brings Haswell graphics support, 3D acceleration, and improved performance in extremely high-traffic networks. DragonFly now supports up to 256 CPUs, Haswell graphics (i915), concurrent pf operation.
The latest version of DragonFly BSD no longer supports 32-bit x86 machines and is designed to work exclusively on the 64-bit x86 architecture.
What is Dragonfly good for?
I mean that genuinely? In today’s world of easy virtual machine / container provision and scaling … with application development frameworks now making it easier to horizontally scale …. in the world of distributed replicated resilient storage …
What do people use Dragonfly for?
What#s the point of Haswell graphics if it is supposed to be a headless server .. or a deployable image…
Does it do something better than MacoSX -> Docker/AWS / GoogleAppEngine -> S3, HDFS, ZFs … ?
Maybe it is the best at concurrent multi-thread/process workloads?
Maybe it has the best network stack? Highly parallel packet filter? Maybe it has the best network IO throughput (or latency?)?
I love open source, and diversity – it’s just sometimes I find it hard to work out which job a tool is good for!
One of the links in the summary points to the project’s feature page. That should answer your questions about DragonFly BSD and what it is best suited for.
The goal of this OS is to have the best possible performance. As far as I know, this goal was reached having as few locks as possible in the kernel and providing several optimizations on concurrency (Light weight kernel threads, for example).
As far as I remember, the DragonFly leader implemented several years ago a DragonFly usermode kernel, very useful to debug the kernel using common debugging tools and also useful to launch “virtual DragonFly machines” in user mode. In this virtualization/elastic world I imagine such feature very useful.
Edited 2014-11-27 00:57 UTC
ebasconp,
I haven’t looked at the BSD kernels, so I’m curious if someone can speak to what else DragonFly BSD has done specifically to optimize performance.
Having lightweight threads is good, but I find the best way to optimize performance is actually to use threads less and promote asynchronous calling instead. This way activities that “block” don’t need spin up any threads just to wait for blocked events. And when we don’t block, many synchronization points can be done away with entirely.
If I were designing a kernel (again), I’m not sure I’d have a need to support kernel threads at all – every request would go in, modify some state, and go back out without ever blocking. Think about it, why should a piece of kernel code ever block? Instead of blocking a thread over disk IO, you can just register the request in a FIFO and exit without locking a thread. Technically kernel pages might block via the VMM if they’re swapped out, however we could peg kernel into RAM. Each core would have a kernel running on it, but with no threads we solve a whole sleuth of common MT problems. Threads are fundamentally needed to solve the problem of multitasking long running CPU intensive tasks, but when do we have these kinds of tasks running in the kernel? I’m not aware of any and in any case they could be delegated to a userspace process.
Edited 2014-11-27 03:47 UTC
How exactly would your kernel assign HW/OS resources without kernel threads?
tylerdurden,
I’m not sure I understand the question, but I’ll do my best to describe it.
The kernel gets a request to do something, lets say read from a socket, the kernel could use a thread and block on input, but why do that? The kernel is going to need to register a callback on the pipe to wake up the thread anyways, so instead of waking up the thread inside of the callback, why doesn’t the kernel perform the same state manipulations and callbacks but do so without using a blocked kernel thread. This way the kernel proceeds to handle other events without giving up a thread to blocked IO. The userspace caller (assuming it was a blocking call) can be set to the same blocked state regardless of if it’s waiting on a kernel thread or a pipe event. It doesn’t care that a kernel thread was blocked to handle the request.
Linux (and probably FreeBSD etc) don’t use up a kernel thread when reading from a socket. However other types of IO, ie file systems, are built on top of blocking kernel threads. In linux (and probably BSD) Threaded filesystems were the convention since the beginning, redesigning all filesystems to be asynchronous at this point to eliminate the threads would probably require the biggest patch linux has ever seen…so it’s not really on the table.
Anyways, since threads are notoriously difficult to cancel (ie it may be holding locks), a serious consequence of this is that sometimes the blocked threads cannot be canceled, which is why to this day we can see NFS/SSHFS lockup. Sometimes we see processes that are blocked on IO against a dead NFS endpoint – it just hangs and cannot be killed until the network IO times out. Non-cancelable kernel threads are why it happens.
This, and the complete elimination of blocked threads/stacks is why I’m a proponent of async design. Modern servers like nginx use async for these reasons.
Edited 2014-11-27 05:08 UTC
What do you think of the Ada tasking model for such a thing?
kwan_e,
Although I read about it’s features from time to time, my working knowledge of Ada is pretty poor.
http://www.tc.umn.edu/~puk/Adatasking-Pukite.pdf
https://en.wikipedia.org/wiki/Ada_%28programming_language%29…
I like the structured way that Ada declares tasks, but it still looks like an Ada task is a non-preemptive thread plus stack. If the task blocks before running to completion, then I would think the stack remains occupied by the blocked task until the task can continue. So unless I’m mistaken, I don’t think Ada tasks can eliminate threads. It would be nice, but it would have to be pretty sophisticated.
I initially thought coroutines might be used, but that doesn’t seem quite right because our problem doesn’t fit the normal producer/consumer pattern on one stack. Co-routines have a special relationship, but async OS requests are mostly unrelated/unsynchronized.
I’ve implemented the ideas in a linux library, which is functional. Maybe I can pull out examples, yet it’s a very weird design due to the way in which allocations are handled to minimize dynamic allocation. The resulting code is efficient, but not very conventional. C++ makes the unfortunate assumption that instantiation and allocation happen at the same time. I was thinking of ways to abstracted everything behind a pretty OOP language before getting sidetracked.
Maybe DragonFly BSD would be a better home for this stuff. I don’t know if DragonFly supports true async File IO, but I’ve never liked how posix AIO support in linux is accomplished using threads to work around kernel limitations.
Edited 2014-11-27 08:10 UTC
I found these, don’t know if you’ve tried them. I certainly haven’t:
http://en.cppreference.com/w/cpp/memory/get_temporary_buffer
http://en.cppreference.com/w/cpp/memory/raw_storage_iterator
Placement new allows instantiation in storage that was previously allocated.
http://en.cppreference.com/w/cpp/language/new
But in order to support blocking IO the kernel would have to spin up threads anyway wouldn’t it? Are you saying you would abandon the concept of blocking calls entirely?
Not saying it isn’t a good idea, but like it or not the vast majority of code out there uses blocking io simply because it is easier to reason about. That isn’t meant as a criticism, just an observation.
Out of curiosity, what mechanism would you use to re-enter execution? Would you try to stay POSIX compliant or are you interested in going your own way?
galvanash,
Yes, because I don’t think we need them in a kernel.
You are right, but I look at it this way, programmers will use what they are comfortable with until they have a reason to adapt. Threaded designs are still popular in common programming, however async is rapidly closing the gap in systems like webservers where scalability matters.
Even if it weren’t for the scalability benefits, I still prefer async design, because I find the code much easier to write correctly than MT code. MT code often exhibits notoriously subtle race conditions.
While there are improvements that could be made to the interfaces, the idea here was just about using AIO within the kernel, not to change the userspace interface (ie POSIX).
So for instance, a userspace thread does a standard blocking read request. A sysenter switches to kernel mode. Assuming the kernel is limited to one thread per core, then this thread MAY NOT block (otherwise it will implicitly cause CPU bottlenecks).
The kernel tries to retrieve the data from cache, if present, return the request without blocking.
Now the userspace thread MUST block, so set it’s status to waiting.
The kernel sends the read request to the fs/disk IO queue and register a callback to complete the userspace syscall upon data retrieval or failure.
Now the kernel is done servicing the syscall, but it cannot return to the blocked caller, it looks for any unblocked userspace threads and runs those instead.
Now the kernel gets a disk interrupt telling it the data is ready, it looks at the callback table and performs the callback for the syscall that invoked the read request. Now that the data is read the original userspace caller can be unblocked and it’s execution can resume (right away or the next time it gets scheduled). In this way the userspace can retain it’s original posix/blocking semantics while the kernel handled the event completely asynchronously without the need for threads.
Does this answer your question about the “mechanism”?
Edited 2014-11-28 19:27 UTC
Ok, I think I follow now. Your talking about stuff way above my paygrade though
Your basically saying you want to make AIO handling use a true in-process state machine in kernel instead of using a kernel threads as a crutch so to speak. That the idea?
Anyway, I use node.js quite a bit and I am a fan of async designs. Sounds neat.
galvanash,
I feel underpaid :-\
Yes. I’ve done it in userspace, but I always felt it would be good for a kernel.
Do you have time to work on side projects? I could use your help
Wish I did. I barely have time for my actual job…
Out of curiosity how does their HAMMER FS compare with FreeBSD’s ZFS?
This gives a nice overview:
http://www.bsdnow.tv/tutorials/hammer