I’ve always been curious about how things work. When I was little, I annoyed my parents with millions of
questions. Why is the sky blue? How does water come out of the tap when you open it? Maybe I was born without the
mental switch that lets you be happy with using something, without trying to figure out how it works.
As I grew up, I became interested in writing software, first as a hobby, then as a professional career. Naturally,
unhappy with writing applications on top of a black box designed by someone else, I wanted to work on operating
system development, and advance the state of the art.
I had written real-time monolithic kernels for the
Motorola 68000 and the
Intel 386
as a hobby, without any new idea in
them. I started thinking about designing my own operating system around 1996. Before I looked on the internet, I thought
I was the only one with such a preposterous idea. After all, the bases were pretty well covered.
Windows had a lot of problems, but it got the job done. The power users, or the ones who wanted to feel special, had
BeOS, Linux and MacOS. The server space had all things Unix, and NT coming up to smooth things over. The embedded
space had lots of real-time kernels, none really special, but people made products with them that work in the end. Why
would’ve anyone wanted to invest time in OS development anymore? I looked online anyway.
I was shocked to find a LOT of projects.
Literally hundreds. I was thinking I should stop right then.
Certainly, out of those hundreds of projects, someone was already doing what I was thinking about, and there was no
point in duplicating the effort. I looked further and was surprised to see that all of them, except for a couple known to
OSAlert readers, were stuck at “dream stage” or “bootloader stage”. Most of the time, the creator was gone or idle,
replaced by another set of equally inactive people. Sometimes the sites were
flat out unmaintained for a couple of years
already. Online ghost ships.
Stuck in dream stage meant someone decided to create an
operating system that would run Windows,
Linux and MacOS applications all under one architecture, created a site, and explained how this is really a good idea that
nobody has done before. Nobody had seen the opportunity. Then, the person realized how complex the
dream was while being pointless, and left without a forwarding address.
Stuck in bootloader stage usually was expressed by a site starting with a goal such as, the next generation
microkernel-based operating system. The site offered a bootloader, the small piece of software responsible for loading
the kernel, for download. Nothing spectacular. It would read the kernel and jump into it, except there would be no kernel
to be found. Somehow the author thought that if they could write the bootloader, the kernel would be a piece of cake, or
something. Then somehow disappeared.
In this context, the few exceptions were like Las Vegas, a city of lights in the middle of a desert. I’m thinking of
AtheOS and SkyOS.
Today you can add OpenBeOS to the list. I personally hope
we can add Syllabe to it soon.
How come they have a working system where others are stuck in bootloader land? Are they smarter? Do they
know something the others don’t? Well, considering AtheOS was written by one guy during weekends and leisure time
over the course of only four years, the smart part probably doesn’t hurt. Let it sink for a second: One guy.
Hobby. Four years. Full kernel. Quite a few drivers. Graphical interface. Applications. One guy.
Okay, let’s proceed.
Back to the story of my life. I decided to go ahead with my project. I met the right people for it in 1998. We decided
the idea was new enough to deserve a company to develop, market and support it. Probably it would be closed source,
as it is now. The goal from day one was well defined. Today, we have a full OS, graphical interface, telecom stack, and
real world applications, portable across several processor architectures. Around fifty people bang on it full-time. To
paraphrase
Joel Spolsky, our marketing department
goes out to say that it cures baldness and heart attacks. If you buy
two, neither is free but you get a really cool pen. All that really matters is that customers say it works.
Projects that work out to completion share common characteristics. They are in no way a recipe for success, but
they form a prerequisite.
If you’re thinking of starting, or working on, a new OS, not making decisions will leave you with a disaster. A site
without a forwarding address. The umpteenth bootloader. Yet another TakeThatLinus!OS that aims to take over the
mindshare of all open-source developers.
You can argue that you are going to work on it as a hobby, so I should shut up and go back to my company. What
the hell am I doing writing this on a Saturday, instead of making plans for my business, right? Once you make
these decisions, your project will be a lot more fun to work on. Taking out the frustration early on will make a much better
use of your free time, and if we’re lucky, we will be granted with a new project with the quality of AtheOS. It will be
pointless for most of humanity, but they don’t read OSAlert in the first place.
I will not offend you by mentioning commonplace ideas in this article. You already know you need lots of time, lots
of patience, lots of datasheets and books, and the capacity to debug with your hands tied in your back.
Choose your own camp
You need to decide early on what category you are in.
Are you a low-level hacker who wants to write a fully working OS
in assembler, because those damn C people don’t
see how much speed they lose?
Do you want to learn how kernels are written, by copying an existing design (unix, whatever?) or
make up your own
from a hodge-podge of concepts? That seems to be the trend lately. Or a convenient excuse.
Do you want to advance the state of the art? Create a new architecture with a lot of advantages nobody has seen
before?
Pick one, and stick to it. Software hates inconsistency. In the middle of the project, don’t decide to
rewrite
everything in C because assembly is a bit much to chew after all, or to publish papers on something everybody already
knows (like, say,
programming the PIC on x86) just to show the
world you can read a datasheet.
Choose your audience
Who is your user? Not the fellow hackers you will hire for help on coding your dream. But who is the end user?
Billie, a middle-aged house wife with three kids, who does the accounting around the house, and also chats a little
bit online once in a while when everybody is at work and at school?
John, a thirty-something guy writing unix banking software for a living, currently using NetBSD on his desktop, switching
on a regular basis, trying out FreeBSD or Linux, because he needs the unix environment and will not mind a new OS as
long as he has gcc, nedit and windowmaker?
A person known as DaMan, a 21-year old sysadmin/programmer who compiles every new linux kernel on his company
servers, as soon as he hears they are out on the kernel mailing list? And who will be willing to try any new cool server OS,
as long as it has perl and vi?
Fellow hackers in the OS development community, who will eagerly read your papers about thread migration and
protection through artificially software reduced instruction sets?
Yourself, who will be the only one to ever see your OS?
Be honest about your goals
When you know your camp and your end user, the killer question is: can you do it? Be honest with yourself.
Give yourself credit – you will never know if you can write a fully debugged preemptive scheduler, until you have tried it, but
be realistic about how much you can do. Either you will decide it’s more fun to play cricket, or you will have a better idea of the profiles of people you need to enroll on your site. Either way you win.
My company is happily employing a guy best known in the open-source world for writing a mono-tasking OS all in
assembler. It was limited, and for x86 only, but hell, it worked, and it fulfilled the original goal. One day he told me he
received an email from a fan, who said something like: “I just want to know how to decompress the kernel, because I can’t
figure it out. But the rest, the writing the kernel thing, that’s easy. I just want to know about the unpacking”.
Right.
Publish your goals and your own capabilities with honesty. Smart people don’t like arrogant people anyway.
You are the god of your project
If you go the open-source route and enroll fellow hackers, they will come up with their 130 ideas they want in your
kernel, plus their 200 itches to be scratched (such as “semaphores suck!”), and their own idea of your project. If you let
people hack away merrily, you will end up with the opposite of efficient.
You create your project – you own it. Be sure that contributions are consistent with your vision. Don’t hesitate
to ask people to rewrite something so it fits your goals and your architecture.
Do not bait people on purpose, though. Make sure that as much as possible is written and published on your
site. You can’t bash somebody because he doesn’t know something is not in line with your thinking. If they’re good, they
will move on to another project. You don’t want to lose good people.
Version control is great and mandatory. Letting anybody commit code, no matter how focused and smart they are, is
the best way to have your project go in all directions and try to be a distributed server OS as well as a RTOS that
fits in 64K, at the same time. Be sure people send patches to you and you commit them when they are OK. After a
while, when you have grown to trust a few people like yourself, expand to a core team with commit privileges. Make people
own the task of integrating, validating and committing.
Conclusion
If we go back to AtheOS, the goal of learning and cloning was well defined. The target is the power desktop
developer-users who like to try out cool new things. Kurt, the creator, never claimed to dominate the world and has
been going at it piece by piece, slowly. He did most of it by himself,
not encouraging external contribution, enforcing
consistency.
Taking these decisions right away also helped me a lot.
If you’re engaging in such an adventure, I hope this helps take the frustration out and make it much more fun. In any
case, good luck, and I would love to hear comments on the article, as well as experience of anybody who already went
down that path, successfully or not.
About the author
Emmanuel Marty is the Chief Technical Officer of
NexWave Solutions, the supplier of a new software
architecture, OS services, and telecom stacks, for consumer electronics and telecom. He has been working with
computers since the age of 10. Currently aged 26, he lives in Montpellier, France, with his fiancee. He can
be reached at [email protected].
Most people write their own OS to learn. Others as a hobby. Not all OSes are commercially viable, not all OSes are actually good. But they do serve one purpose, education. That means real-life experience of designing the kernel, design this service, that daemon and so on.
In fact, linux was created for educational purposes. Same with AtheOS. I could list a lot down, but really, I think you get the idea.
Not everyone has a new idea. Not all OS in the first place could succeed. Take Be OS for example.
Yes, I agree. Most of the new hobby OSes are for educational purposes and for the fun of coding late at night and ordering pizza. It’s fun.
I have never tried it, so I would have to take your word for it. But in this part of the world, with pizza, it is quite an expensive hobby.
Hi Rajan….you could always try rendang sapi and Tiger Beer…..but then you probably wouldn’t get much coding done!
Perhaps there is something deeper than just education, entertainment, and extra thin crust pizza?
You’ve felt it all your life. Your sense of powerlessness. It is what drives you to create your own operating system. So you can be in control. Or more likely, so you can feel like you are in control.
For a moment when your creation is running — living — you feel a sense of power, of freedom. You are building something free from the fetters of the outside world.
You have to put your creation aside, yet you long for that feeling. You create reasons to go back to your creation. Oh yes, the mind can create story. Humans have no shortage of “reasons”. Untrained minds can’t live without them.
Follow the white rabbit.
#m
Sometimes, I think that if computers hadn’t been invented, all of these kernel hackers would be in their garages, creating amazing new engines for their cars.
Sometimes they’d be faster, sometimes they’d be more fuel-efficient, sometimes they’d be elegant but impractical, and sometimes they’d just blow up.
But it wouldn’t matter. Nobody would be knocking them because they were creating engines that couldn’t knock GM out of the marketplace, or that they weren’t polished enough to be daily drivers. Their interiors might be ugly, and they wouldn’t necessarily be comfortable, and they might run on pure alcohol or vegetable oil — but the engine hackers would be having fun, and swapping knowledge and stories with other engine hackers, and happily driving around on the same streets as the minivans and the corvettes with a sense of pride that ordinary drivers lacked.
And once in a while, a Smokey Yunick (the closest thing to Linus that I can think of) would appear, and create something that made everyone sit up and take notice. The other engine hackers would look at their work, and use that knowledge to improve their work, and…
…ah heck, you get the idea. Run, boys, run — get those hot rods running, and let us drive ’em around once in a while.
Very interesting. One can conclude from two of the previous posts that the field of computing is absorbing the creative talents of those whose natural ego-drives would otherwise be contributing great works to humanity in other fields. I started out wanting to read the article for inspiration about writing your own toy OS, but now I just feel stupid for sitting in front of a monitor when there is a whole real world waiting to be experienced and interracted with just outside my door…
The problem with Operating Systems is that they’re complicated. Once you have a bootloader in place, you’re faced with the prospect of having to create a working kernel, and that is where everything suddenly becomes a lot of work.
A lot of people want instant results; they want to see a process running, they want to read and write to a harddisk, they want to draw windows on the screen…but it doesn’t work like that. Kernels are big, and interdependent. You have to write almost all of the kernel before you can even compile it! A lot of people get disheartened right around the stage where they realise that they’ve got 40k of source, and they havn’t seen any of it do anything yet…
He did most of it by himself, not encouraging external contribution, enforcing consistency.
Well, that had some advanages, and a disadvantages. The advantage, as you state, was that the AtheOS base distribution is consistent.
The disadvantages? Well first of all, that consistentcy is only skin-deep. Because there was no standard way to do E.g. ImageButtons with the base libatheos, almost every application developer crearted their own. The resuly is a mish-mash of styles and images, and its ugly!
On top of that, everyone else is dependent on one person for doing all of the work. Good ideas go to waste, because the developer doesn’t have the time to keep track of them all.
Last but not least, it created a lot of confusion in the AtheOS community. Some patches would be accepted (Wheel-mouse support, ISA DMA), and some drivers would make it into the base (The nVidia driver). However, a lot of people never sent their work to Kurt, as they thought that he didn’t want any contributions. It can be very frustrating to see a lot of good work left out on the side, like an abandoned child.
If you go the open-source route and enroll fellow hackers, they will come up with their 130 ideas they want in your kernel, plus their 200 itches to be scratched (such as “semaphores suck!”), and their own idea of your project. If you let people hack away merrily, you will end up with the opposite of efficient.
You create your project – you own it. Be sure that contributions are consistent with your vision. Don’t hesitate to ask people to rewrite something so it fits your goals and your architecture.
One tool we’ve found to be a great help with Syllable has been a Wiki site. It allows us to keep track of ideas, discuss them, offer alternative solutions etc.
Its much easier than dealing with a bunch of emails, and everyone can easily follow the conversation. Once you have a solution, its there, on the page, and you can refer back it whenever you want.
Hopefully I havn’t scared anyone off though
Go and write your own OS, and learn as much as you can. Just don’t be under any ilusions that it will be easy, or instant.
I agree, many “little, alternative” OSes have been written for educational purpose (that’s why many of them are relased under Open Source licenses).
What i can’t understand is why this knowledge is *not* shared.
Well… i try to explain it better:
– Many of this OSes dies quickly due to development lacks;
– one developer os have no future;
– knowledgle (books, docs, etc) are easy to find, but too mutch scattered on the net;
We (me, qStone and K J) started an ambitious project: create a great OS Development Portal for all open source operating systems, in order to help developers in collabation, sharing ideas, techniques, knowledge and anything they needs. A “SourceForge-like” portal but devoted to Open Source Operating Systems.
We hope this will help all the growing OSes to become usable & mature.
To reach this goal we need PHP skilled developers, web designers & surfers.
If you’re interested, do not hesitate to write me.
OS Factory forum: http://more.at/osportal/
I remember this, I followed the hype of that project for some time.
Started by a 12- or 14-year-old so-called “boy genius”, It began in 1996 in its “early planning phase” and died in 1999 or 2000 in just the very same “early planning phase”. In-between the claimed that “hundreds of developers” were involven with the project.
The boy-lead promised a better Windows than Windows, a better MacOS than MacOS etc. etc. – the only thing we got in the end, though, is (hopefully) a more humble young person
Chris
Unununium development didn’t stopped because the team wanted to rewrite everything in C, the vision and the target still is a full working os in ASM. The reasons of the discontinued development are explained in UUU page. Assembly never dies, it resurects!
Emmanuel, Like Andreas just pointed out, Unununium development wasn’t doomed because of a rewrite in any language, and surely not in C. Uuu development was closed because of the lack of talented coders or dedicated ones.
We were at some point close to 30 developers yet only 3 of us committed code on a regular basis, the others just hanged around and looked ‘l33t’.
While the main site and source tree stop’d moving forward, there was still development being done in the background in some dark cranny of the universe. And to continue on with Andreas’ note, Assembly never dies, it resurrects!
A new hope… A new beginning – UUU
mhm so i go check out the os portal site. wow. i get some board statistics, and nothing else. its like a message forum without the forum! cool. how helpfull.
in another note, the os journal site keeps moving forward ( http://www.cyberscriptorium.com/osjournal/cgi-bin/index.pl?action=n… )
as well as the os xml site ( http://osdev.berlios.de/ )
I found the article to be very inspirational because it offers down-to-Earth advice. I especially liked the following points. Please note that I present them as I perceive them myself after reading the article, you would still do well to read the article in full if you haven’t done so.
* Be clear and honest about your goals — to yourself and others.
* Go ahead and try it, you will gain experience and understanding. But also be realistic about what you can do.
* YOU OWN YOUR PROJECT — make sure contributors are consistent with your vision.
Funny, I am transitional about my personal life, and these are the three most important points that I see I can apply to myself, and not just to the software projects I’d like to initiate. Thanks Emmanuel for the insight.
YOU OWN YOUR PROJECT — make sure contributors are consistent with your vision.
I think this is quite significant; most of the operating system projects I know of that have made significant progress have been one-person efforts.
Yes, I agree. Most of the new hobby OSes are for educational purposes and for the fun of coding late at night and ordering pizza. It’s fun.
How sad it is that we actually find this type of work not only interesting but enjoyable. I’m saying that in jest of course, since I find coding these kinds of esoteric stuff fun as well. I may take this article as an opportunity to code up my own OS for the fun of it. Nothing spectacular, just to do it.
PS. Thanks for the last few days of postings on OSAlert…very interesting articles.
Since it was mentioned I looked at it.
The kid (Jason?) should have been climbing trees in a nice little forest, skipping stones in the pond, riding his bicycle, stealing apples from the orchard, chasing girls, lots more fun than dreaming OSs at that age. Life is too short to miss on that really good stuff.
I am really glad that until I was in college, computers were completely off limit, not available except time share, so did I miss anything, probably not, it wasn’t there yet.
So I read the article, well what was said could as easily apply to quite a few engineeing projects.
I wouldn’t be so proud to announce he’s been doing OS design since 10yrs old either, I don’t think that will help get new customers, I want to see educated experience in a supplier, not blind enthusiasm.
Now when I see kids looking for VB or Java books in the CS section of the book store with dad of course, I have to wonder, am I going to let that happen to my kids, I really hope not. Time for a picnic!
Hi Rajan….you could always try rendang sapi and Tiger Beer…..but then you probably wouldn’t get much coding done!
Ohhhhhh, someone close to a Malaysian!
I think Michael has a good thought there. I’m not a programmer but, if I write an AppleScript (or, in the past, put together a HyperCard project) or create FileMaker databases for non-profits and make it as good as I possibly can, there is a tremendous amount of satisfaction that results. As he said, part of it is a feeling of control, that you made something and it works, just as you had it in your mind as working. Also, for example, when I make a database for a non-profit who have been using old DOS databases, the look and even squeals of delight from the people who use them day in and day out are worth it all.
Just a small note,
I agree that there are a lot of boot loaders and dream stage OS’s out there currently.
But neither of those stages is exactly easy.
I see far to many dream stage OS’s done by people who’s aim is to replace Microsoft or Linux or some other major OS.
To come up with a truely original OS requires a lot of effort. It most likely has been done before. Even if not successfully.
And boot loaders are no small thing either. You can use software such as GRUB to get past the boot loader stage but to make your own boot loader requires knowing a fair bit of Assembly language and principles.
It may not be as hard as the kernel but I know that when I first looked at a boot loader I almost fainted. Since then I have learnt Asm and I can just about follow the code.
My advice to people interested in OS development is to first try looking around at existing OS resource webpages (www.osdev.org has a very nice links page) and to check the newsgroups and IRC channels. (irc.openprojects.net #osdev and #OSDevelopment and the channels for the programming languages you the OS in)
Then when you have got in contact with some people who have been there you will find it a lot easier and you have places to ask questions if (more likely when) you do get stuck. And you know what websites to look at for ideas.
But don’t be afraid of failing, even if you try and fail you will with luck have learned something.
Daniel
Is http://tunes.org/Review/OSes.html ever going to get updated?
JJ, I couldn’t agree with you more. I’m glad i barely did anything with computers till college. I know they have sucked more time/life out of me than I should have let happened. I hope this is not where everyone is going. Then again maybe we have no choice. In this day and age. The DEP would get your for skipping stones in ponds, the Parks Service for climbing a tree. Go to jail for stealing an apple from an orchard, and get a sexual harasment suite for chasing a girl, and riding a bike on a 8 lane highway is unsafe.
maybe inside is good. Just get them into the history books instead of programming.
Get the deep knowledge of existing OS designs.
More often then I like to see people are willing to make an new OS just because “<OS-de-jour> sucks!” . Well it may or it may be not. Windows NT was designed by former DEC system engineers with a lot of background in kernel development.
If you want to develop something better you need to be at least on par with those guys. True, a lot of water had flown since DEC R&D published their papers, but there is enough read about OS kernels to keep 14-years-old preoccupied till he/she gets his/her beer drinking permit.
Linus was college student when he started and he used Tanenbaum’s book as a guide. That’s a good approach for OS developer but it’s not new anymore.
Strange that nobody quoted JLG on the subject:
“I worked 22 years in the industry, and I noticed that operating systems get cancer with age.”
And there was one another about writing multi-threaded drivers and growing hairs on your chest I cannot find on begroovy.
> Just get them into the history books instead of programming.
Get them into thinking, analyzing and doubting everything. That way they’ll be a good programmer and a good person.
I had to learn this myself, but you can teach your kids that.
… you can try the Flux OSKit:
http://www.cs.utah.edu/flux/oskit/
It’s a neat little tool, and it lets you concentrate on the meat of your OS project without worrying
about the little details.
Peace,
TLFord
Unless things have changed, I wouldn’t recommend the OSKit…no offense or anything…it’s a neat concept…but unfortunately there are a lot of problems with OSKit…releases tend to be few and far between…documentation is HORRIBLE…and last time i used it (Cinco d’ Mayo release) a lot of things were terribly broken…i think you’d get more milage by modifying an existing kernel…
just my $0.02
-bytes256
Really? That’s odd – I’ve been using it for the last couple of years, and it’s been pretty cool. I’ve gradually swapped out functionality in their native libraries with my own code, though. What parts are busted – I’ll be sure to stay away from those modules
And thanks for the info – I love it when I find out something has potential problems. Saves me the trouble from finding ’em myself.
Peace,
TLFord
I think for an os project to be a success in terms of being adopted like Linux; several things are needed. Not all were mentioned in the article.
1. Devoted moderator with good coding skills.
2. Right timing.
3. Addressing some kind of common need.
Linux had Linus. It went public same time as internet started gaining popularity. It addressed need for a zero fee license os, needed by companies that were built on business model requiring large number of boxes being deployed.
AtheOS had right moderator, but it never made it because other “reqs” were never met.
That’s just my opinion, so feel free to disargee.
To all,
Thanks a lot for your comments. Positive or negative, peer review is always great.
This editorial piece is about sharing my experience, if it helps take out some of the frustration. Make it more fun to develop your OS and see it work in the end.
It does not even try to cover the commercial success of an operating system. That’s another problem entirely. It’s a conflict of interest with having fun developing something according to your own dream, anyway. It’s fair to say that Linux had technical success as well as being adopted because it filled a need. It’s true AtheOS was successful technically but not for adoption. Both of them do fit in the pattern the editorial piece describes.
About the integration-validation part, it’s true that if you have only one guy, like on AtheOS, his window of time is fixed, and you can either do one thing well and deeply, or many things superficially; what Vanders calls “skin-deep”. What you probably want to do is: design and code the basics by yourself. Later, switch to a position where people send you patches, and you OK them and commit them. Once you’ve grown a team, trust one or two more people to commit as well, then you can go back to a little bit of coding.
Using wiki to avoid losing patches and speeding up their handling is a good idea, Vanders.
Overcoming the ‘kernel fright’ (people stuck in bootloader land) is an interesting topic too. Using GRUB on PC-AT is a better starting point, certainly, if you want to develop your kernel without being dismayed by fitting assembler in 512 bytes and switching to protected mode. As someone said, that will not prevent you from having to know assembler and the IA32 in-depth; but it can get you up and running quicker.
The OSKit is quite targetted at the people who want to advance the state of the art, in a particular domain. They want to implement the VM techniques they describe in their PhD. They want to implement a new scheduler. People who want to learn how it all works will want to start from scratch.
The “top-bottom” approach prevents having to write the whole kernel before you can print “hello world”. You can design and implement a lot of it as an application on top of another OS. Provided there is a good and clean design, moving it to native afterwards can and has been done.
Emmanuel
Brad, you are SO right
Code.
I thought this link should be mentioned:
http://www.acm.uiuc.edu/sigops/roll_your_own/
It was an attempt to make a “how-to” for writing an OS. Never finished, though.
> Never finished, though.
He he. Given the gist of this thread that’s quite amusing.