Submitted by Apparently, Matt Dillion has decided to roll his own filesystem for DragonFly. “Here is my initial outline of the filesystem design. It is open for discussion. Please feel to ask questions for anything you do not understand. I do not intend to start coding anything for at least two weeks. There are currently two rough spots in the design. First, how to handle segment overflows in a multi-master environment. Such overflows can occur when the individual masters or slaves have different historical data retention policies. Second, where to store the regeneratable indexes.”
lets call it that
OS/2 HPFS
rather than flogging it as yet another general purpose OS.
Sorry Matt, as much as Dragonfly sounds and looks cool (I tried v1.3) you’re not going to find any users for your OS given Linux, Solaris and FreeBSD.
OTOH, if you can convince Tivo or Nokia to use Dragonfly, that would be cool.
They are NOT positioning DragonFly as a general purpose OS. They are instead designing an multi-node clustering OS that scales in number of nodes without impacting the overhead of maintaining cache coherency across nodes. It is also suitable for single-node systems (i.e. with only one cache coherency domain), but this is merely a degenerate case of the general design.
DragonFly is designed primarily with upward scaling in mind. On a single-CPU system, the locks in traditional *nix and the MPI in DragonFly compile away to essentially the same model of process and memory management. In this case, you go with whatever has the best hardware and software support, and that isn’t DragonFly.
EDIT: After reading more of this thread, and I totally disappointed with the display of ignorance. Feel free to ask questions if you don’t understand what DF is all about, but don’t dismiss what they’re trying to do if you have absolutely zero clue as to the goals of the project.
Edited 2007-02-25 05:52
This is too silly for words… I mean, it is well known that the real limitation in OSS projects is the available amount of menpower (number of people that have the skills to do the job, and time to work on it).
And this guy is a driving force behind rewriting the core of an operating system. Work that requires experience, insight (foresight, if you will), and in general: solid coding skills. There are very few people that can do what this guy can, AND have the time/desire to work on this particular project.
And then, instead of focussing on the groundwork, this guy says: “I need a new filesystem!”.
Helloooo! There are so many to choose from… Most of them decently engineered, debugged and mature. What does he *really* need that *no* existing filesystem provides? If so, why not pick what comes closest, and add [desired feature] to it?
Come on Matt, don’t get sidetracked, don’t waste your time on stuff that can be done later, or by others. Don’t you want DragonFly to be a solid OS first, before spending time on [desired feature #xyz] ?
Ill be honest up front… I have never used Dragonfly nor do I have any intention to at the moment, but I have been interested enough in it to read up on its status now and again. And I find your remarks somewhat puzzling.
Dragonfly started in 2003 (4 years ago). The goal was to (a) take FreeBSD and implement what the developers felt was a more sensible approach to SMP support with the purpose of laying the foundation for… (b) native clustering. That has been the stated goal since day 1.
(a) is pretty much done now, and (b) is more or less dependent on a new file system according to everything I have read about their clustering approach. How you think this is getting sidetracked I don’t quite follow…
Edited 2007-02-24 22:40
DragonFly has a very specific set of goals, which include high scalability and transparent clustering. From the feature set of the filesystem, it’s clear that its desigened specifically to meet these goals. Read the specs for yourself, it’s a very different best. It doesn’t look like any filesystem I’ve encountered, and certainly not like {JFS, XFS, ZFS, BFS, HFS+, NTFS, EXT3/4, REISER4, UFS2}, which would be the main alternative mature and proven choices.
This is too silly for words… I mean, it is well known that the real limitation in OSS projects is the available amount of menpower (number of people that have the skills to do the job, and time to work on it).
And how is that not “a real limitation” in non-OSS projects?
In fact I’d say that the really successful OSS projects make more progress than the really successful closed-source projects.
And then, instead of focussing on the groundwork, this guy says: “I need a new filesystem!”.
In what way are filesystems “not groundwork”? If you develop an OS w/o a filesystem, what are you going to store your data on – a cheese sandwich?
I always thought Dragonfly was going to use zfs … Why is there need for another filesystem when the really big one – zfs – is on the market???
Because it doesn’t do what he need?
ZFS is indeed a very advanced filesystem. However, filesystems are very tightly associated with the virtual memory system through which they operate. ZFS was designed on the assumption that the OS will enforce cache coherency through mutual exclusion mechanisms. DragonFly implements cache coherency through a fundamentally new model of process and memory management that implicitly addresses the issue of cache coherency.
Could DF fool ZFS into believing that memory can only be accessed under certain conditions? Of course, and this is what they do currently. But this is far from optimal, and it prevents DF from scaling efficiently to vastly multi-node systems.
DragonflyBSD keeps going on its focus about pure server/cluter environment improovements, nice to see their new upgrades, while they will be [even partially] ported to other BSDs, especially for FreeBSD which shares quite large part of source code.
While the desire to have a filesystem that meets specific needs and the ability is admirable, this seems a bit overreaching. It took SUN years to produce ZFS and they had a lot of very capable, intelligent people work on it. It will be years and years before this is production ready, high performance, and reliable. I suppose that it has to start somewhere though, and I hope they succeed given the unique characteristics of the proposed filesystem.
I hope that someone that is part of their project is willing to port and maintain a more mature filesystem in the meantime (such as ZFS) that users can rely on and that doesn’t have the shortcomings that the version of UFS included with DragonFly has.
previous posters who are asking why existing filesystems are not being used .. are missing the point. the existing filesystems don’t meet the specific requirements and that is why a new filesystem is being considered.
one such specific feature, for example, is the multi-master operation. read the linked page for more.
the aims are noble. and the most difficult parts of doing a clustered OS is the data/state – hence the filesystem/memory is going to need attention – and likely overhaul.
how did plan9 pull it off? versus this?
Great question! The basic design philosophy of Plan9 is to extend the UNIX philosophy of “everything is a file” to its logical conclusion. This is a much more filesystem-oriented approach to clustering than is being attempted by DragonFly. While DF began by designing an MPI protocol for synchronizing the process and memory managers, Plan9 began with the design of the filesystem and took it from there.
The Plan9 filesystem is called 9P. It supports mapping any kind of resource–be it physical memory, files, I/O devices, or network-attached instances of all of these things–via a hierarchical namespace that may be unique for any process on the system. It also supports union directories, which is essentially a mount point that may refer to any number of mounted namespaces and resolves reference conflicts in a deterministic fashion.
To be brutally honest, Plan9’s design is cleaner and more elegant than DragonFly’s. Plan9 was destroyed by Lucent Technologies, and it doesn’t have the basic ingredients to thrive as a community-based OSS project. DragonFly’s BSD heritage has complicated their design, but it has helped them cultivate a development community.
Who said it was dead? If you look carefully you will see that in many aspects plan9 has influenced more than you may realize in terms of system design.
I never said it was dead, or that it hasn’t been greatly influential. One could argue that DragonFly’s goals might not be so ambitious if not for Plan9. Plan9 continues to be developed as an open source project by current and former members of Bell Labs and MIT research teams, and there are direct descendants such as PlanB.
May I ask some foolish questions?
1. Couldn’t DF implement the Plan9 filesystem?
2. Couldn’t DF evolve to a logical continuation of Plan9? 3. Is it possible without rewritting everything?
4. What are the candidate filesystems instead or writing another one?
Thanks for your answers.
They’re not foolish questions, but the answer always comes down to “In theory yes, but Matt won’t.”
I’ve known Matt for 10 or 12 years now; since he was at Best inc and I was using it as an ISP. Once he has decided that a piece of software should be a certain way there is no changing his mind.
He’s got a certain model of scalable distributed computing in mind. Dragonfly will meet that model and it’s not Plan 9’s.
Actually, Plan 9 was destroyed by Rob Pike. Inferno was his baby, and when AT&T was splitting up he tried hard to make sure that Bell Labs would end up with Lucent rather than the other potential choices. Rob believed that Lucent, being in the teleco business, was the right place to develop Inferno into embedded devices.
I don’t know about the innards of every filesystem ever written, but there are some really neat ideas here.
* Inherent snapshots (versions of every file are saved until space is needed)
* Distributed records. (all information for the files are stored in a segment near the file) which means the master records can be recovered from scratch by reading the disk
* Other cool things
I say bravo for coming up with novel ideas, and I can’t wait to see how it performs.
And I haven’t even reached the end of the document…
Can you (for the sake of your grandchildren) put 64 bit timestamps in the filesystems? atime, ctime and mtime?
Thanks. And pad them until the seconds since epoch will be switched over completely to 64 bits as well..probably a week before the Y2K like problem.