Let me begin by telling you a little story. Some time ago I needed to run a script at work once a day. We had tons of machines ranging from big Unix servers to Linux desktops. Due to various reasons the script could only be run on a desktop machine. However using cron was disabled on desktops. All other machines allowed cron.
I contacted the IT guys and asked them to allow cron on desktop
machines. A long and arduous battle followed. Among the issues said to
me was that allowing cron on desktop machines would be a security
issue (apparently because then people would actually use it).
When I replied that cron is freely usable on all other machines I got
a bunch of other comments, including a “helpful” way to work around my
problem with a complicated combination of xnest and ssh port
forwarding.
All I wanted was to get cron working. Eventually I was told that
“you can code your own cron program with Python really easily, just
run it under screen and you’re all set”.
End of line!
I had come face to face with the workaround trap. I’m sure
most readers have faced some mutation of this phenomenon where an
obvious problem does not or will not get fixed because you can hack
around it somehow. Workarounds by themselves are not a bad thing.
Problems arise when workarounds prevent the so-called real (or
“correct”) solution from emerging. This problem is especially
prevalent on UNIX type systems, which have traditionally had a strong
culture of DIY whenever the basic tools don’t do exactly what is
required.
Some examples might illustrate the nature of the workaround trap.
The first one deals with binary relocation and the second one with
email protocols.
Where am I, asked the little program
One of the cool things about OS X is that most programs come in
self-contained bundles. Those can be placed anywhere in the file
system and run. Similarly Windows programs can be installed in any
directory whatsoever. In contrast rpms and debs have hardcoded install
locations and simply will not work anywhere else. At least most of the
time.
There are several reasons for this but one of the main ones is that
there is no standardised way for a UNIX program (or a dynamic library)
to ask where is the file it is being run from. Windows and Mac both
have this feature.
Suppose we have a C program that consists of two different files: a
binary called foo and a user-editable configuration file
called foo.conf. These files are placed somewhere in the
filesystem (say /opt/myprog/foo and
/opt/myprog/foo.conf) spesified by the install system. The
user then runs the program, which begins by reading its conf file.
What filename should it pass to fopen to access its conf
file?
Simply passing foo.conf would try to open the file in
the directory the program was launched from (usually the user’s home
directory). Opening the correct file would require some way to know
where the executable file currently resides.
This is where the workaround trap springs into action.
The common solution is that the build system #define:s
a string constant such as INSTALL_PREFIX as the path
where the program will eventually get installed. Then accessing the
data file is simply a matter of string concatenation
(INSTALL_PREFIX + "foo.conf"). This works and is
relatively simple, but the downside is a severe static dependency: the
files must be in the spesified directory or else nothing
works.
You can do all sorts of cool things with relocatable binaries, like
allowing users to install non-official rpms (such as daily
snapshots of Firefox 3) to their own home directories in a safe and
reliable way. But these things will never get explored since there is
no way for a UNIX program to ask where is the file it is currently
being run from.
Note: this is one of the problems being tackled by the Autopackage team. They have a working solution
but it is quite hackish and works only on Linux.
“My great-great-great grandfather used SMTP so that’s what I’ll
use”
An oft-repeated wisdom about design is that you have reached
perfection not when there’s nothing left to add, but when there’s
nothing to remove. Email is one area where this rule has definitely
not been followed.
Have you ever wondered why you need to define both an incoming mail
server and an outgoing mail server? You’d think that a server that can
transport mail in one direction would be smart enough to know how to do it
in the other direction as well. And in fact those server can do that, but
you can not send mail through IMAP.
And why is that, you ask?
Because mail is sent using SMTP. Mail has always been sent using
SMTP. Mail will always be sent using SMTP. To not use SMTP is
unthinkable.
This causes a lot of unnecessary work. Mail client developers have
to code and debug two different protocols instead of one. You need two
different authentication steps, one for incoming mail and one for
outgoing. Basically you need two of everything when one would suffice.
A committee has even developed a way for an SMTP server to communicate
with an IMAP server to retrieve attached files from sender’s IMAP
mailbox so that they can be attached to the current outgoing message.
I kid
you not! I hope they patented the hell out of that 5-phase email
sending process of theirs so that no-one will ever have to use it.
There is a simple solution to this mess. The IMAP server shows
clients a virtual folder called, say, OUTBOX. The client composes a
message and writes it to this folder. The IMAP server takes the
message, possibly checks that it is valid and passes it on to an SMTP
server. Mail sent. With 50 percent less cruft, even.
Sounds too simple to be true? Can’t possibly work? There must be
some critical thing missing?
The Courier mail server
has had this feature for ages. But should you suggest that this
feature be added to other programs, you’ll probably get some blank
stares followed by instructions on how to properly set up an SMTP
server.
Conclusions
Have you ever wondered what Steve Jobs actually does at work?
Looking at his public performances you can clearly see that his most
important task is aggressively seeking and destroying workaround traps
that lie in Apple’s products.
Take for example the iPhone. Almost everything it does has been possible for ages. Telephone exchanges have had the support to make
conference calls since the 80s. However I don’t personally know a
single person that would have used this feature. Mostly because to get
it working you had to enter long magical keysequences with zero
feedback. Smartphones may have this feature in their menus somewhere. On
the iPhone you just push one button. The same holds true for most of
its other features. Some of those features required changes to
Cingular’s telephone exchanges. No-one else was willing to do those
changes because they kept saying “you can already do that by …”.
The free software community does not and can not have one person like
Jobs who would force people to fix needlessly complex systems and
eradicate harmful habits. We must do it ourselves. Therefore I encourage
all readers to think of one example where a complicated and crufty hack is
being widely used today even though creating a more elegant solution would
make everyone’s life easier.
Then post it in the comments. Back your assertion with solid
technological facts.
Because the first step to recovery is admitting your problems.
About the author:
Jussi Pakkanen is a long time Linux user. He has had the dubious
pleasure of beating his head against most corners the personal
computing world has managed to build. The doctors are optimistic
that, given time, he may yet become a productive member of society.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSAlert.
What purpose does posting this serve? This “article” has no point, and I suspect this person isn’t terribly clueful after reading it.
Why be a troll?
On Linux/Unix, makefiles are used to specify which files depends on which files. This is considered like “state-of-the-art” because you don’t have to type in a HUGE command line with all your program’s files every time you want to compile something. And also only the modified files are recompiled. Really a good idea.
Of course, writing makefiles turned out to be almost as bad as tucking the compilation commands into a shell script. So someone invented a makefile generator. And this makefile generator needed a configuration file for each project… (I bet you see where this is heading.)
As programmers used the makefile generator instead of writing the makefiles themselves, they became less comfortable with editing the makefiles, and as more and more non-programmers needed to compile programs on different systems, editing makefiles didn’t look like a good idea any more. Especially with all the different configurations in different Linux distros and on different desktops and servers, which made makefile editing necessary more often. The solution? Make the makefile generator generate a shell script that generated the makefile.
So, instead of writing a makefile (“config file for make”), we now write a config file for the makefile generator generator. Nice improvement.
So why are makefiles needed at all? Partially because of another hack: header files, partially I don’t know why. Pascal does not need makefiles at all. A pascal compiler can automatically find out which files need recompilation and links the object files together automatically.
One command and the library path in the global configuration file is all that is needed to build an entire project. Why does C have to make it so hard?
Check out Ant and Jam — they are next generation Makes
They will still just do what should not be necessary. (I don’t doubt that they are slightly better at doing it, though.)
I don’t think so. Yes, the whole autotools compilation procedure is awfully hackish and better ones are available, but the compiler isn’t a make system and should never be. Even things like Microsoft’s .NET do have a build system, it’s just that it’s part of the IDE and it keeps track of dependencies and such.
Compilers and linkers need to remain separate from each other the build system for quite a lot of technical reasons.
1) People like to swap compilers, languages, etc. and link object files from different languages together. There’s no way a single compiler could work all that out itself and manage it, calling other compilers and such.
2) Large programs have complicated things that need to be done during compilation, such as i18n and compiling modules into dynamic libraries, and so on. Usually for modularity, performance and memory reasons.
There are build systems that can do things like work out dependencies and module build orders and so on far better than autotools. The main thing preventing these from taking off is that there are standard ways of checking libraries provided for autotools (pkgconf files), but there is no ubiquitous standard for non-autotools build systems. Plus you can be practically certain that a GNU system set up for compiling has that build system installed.
For these other build systems to take off, there needs to be a *decent* standard created for library checking and such.
Also, another minor advantage of autotools based things is that the final output isn’t dependent on autotools being installed. All of the pkg-conf stuff is “compiled” in a sense into a single ./configure shell script after autogen.sh (typically) is executed before distribution. Maybe others do this, I dunno.
So, instead of writing a makefile (“config file for make”), we now write a config file for the makefile generator generator. Nice improvement.
This is not limited to C/C++:
http://discuss.joelonsoftware.com/default.asp?joel.3.219431.12
This is part of the reason why people are migrating en mass to languages like Ruby and Python.
yeah python, (same thing goes for ruby) to make your script work you have to write at the beginning : #!/usr/bin/python
but wait, what if python is not located there.
solution : #!/usr/bin/env python
but wait, what if env is not there either !!
what a hack
so just remove that line and type python before the script
This is part of the reason why people are migrating en mass to languages like Ruby and Python.
Most python apps come with configure scripts/makefile to make sure that all required optional 3rd party modules are available and to precompile everything to .pyc anyway.
I don’t think it’s caused by header files, I’d say it’s actually caused by linking issues; and I wouldn’t call this a hack.
Some of the complexity of making a working makefile is finding header files; but it’s not an issue of header files: It still exists in Java for example with $CLASSPATH.
In my opinion the trouble is a lack of separation of concern. The makefile is definitely well suited to building code: Most of us could probably make a makefile to build a medium complexity project for one system. The trouble comes into effect that there’s no effective separation between makefile commands and the shell and so you end up dependent on the OS.
One place you get stuck is installing, and another is the joy of filling in the -I and -L’s.
I think that what we need is a smarter makefile that encapsulate’s our -I and -L’s. Some kind of make + pkg-config (note that what we have is _ok_ but it’s a hack since you need both, and then you need your packages to install .pc files). The beauty is that programs installed with this system would automatically contribute their .pc because it would be automatically generated by the install section!
I don’t think syntax actually needs to change much. But I would prefer to separate make from the shell in typical sequences with some extra keyword to get shell integration (and a big warning about how you’re going to break compatibility in books).
all: target
target { needs gtk+-2.0 }: file1.object file2.object
->link file1.object file2.object as target
file1.object { needs gtk+-2.0 }: file1.source [c]
->compile file1.source to file1.object
file2.object: file2.source [c++]
->compile file2.source to file2.object
Maybe not so many keywords, but I think you get the idea.
Edit: And of course osnews doesn’t help me out and it’s all flat, just pretend it’s indented if you see “->”.
Edited 2007-04-17 20:41 UTC
“I don’t think it’s caused by header files, I’d say it’s actually caused by linking issues; and I wouldn’t call this a hack.”
That’s correct. Makefile controls dependencies (questions if everything or just a part of a project needs recompile), proper linking (here, header files affect) and conditions at compile time (compiling for different architectures, with debug enabled, with optimization, with certain features turned off or on etc.).
“Some of the complexity of making a working makefile is finding header files; but it’s not an issue of header files: It still exists in Java for example with $CLASSPATH.”
The access to header files belonging to the system is usually done via the include/ directories with their default settings, making them available using the #include <…> macro, or calling them absolutely with #include “…”, which should be avoided.
Makefile automates parameter calls for cc and ld, such as -I <include path>, -L <library path> or -l<library name>.
“In my opinion the trouble is a lack of separation of concern. The makefile is definitely well suited to building code: Most of us could probably make a makefile to build a medium complexity project for one system. The trouble comes into effect that there’s no effective separation between makefile commands and the shell and so you end up dependent on the OS.”
Usually, /bin/sh (default Bourne shell) is invoked for every command in Makefile. If you want to place them into more than one line (starting with tab), you need to end the line with a backslash, otherwise you’re running into trouble. Access to shell variables is a bit complicated; while you can access make variables via $(NAME), you need $$NAME for a shell variable which is only active for the current line (representing a shell command call).
I use Makefiles for more than programming, such as for LaTeX or HTML builds of larger document projects. Surely make is not inteded for this kind of work, but it’s quite nice for such kind of work, as long as you know what you’re doing.
“One place you get stuck is installing, and another is the joy of filling in the -I and -L’s.”
This is a problem in fact. You mentioned some suggestions. Today, the Makefiles for the FreeBSD ports collections serve this purpose. They do not only compile programs, they also:
# make update
# make fetch
# make extract
# make patch
# make uninstall
# make reinstall
# make install
# make package
# make clean
You surely know what this does in reality. It works fine with the ports, but applications not included in the ports usually do not have this functionality.
“I don’t think syntax actually needs to change much.”
I’d like to get rid of the tab character, instead, having some kind of grouping mechanism as in C, just as an example:
foo.pdf: foo.tex bar.tex blah.eps {
… commands here…
}
“But I would prefer to separate make from the shell in typical sequences with some extra keyword to get shell integration (and a big warning about how you’re going to break compatibility in books).”
So make must be able to realize the command content independent from the actual shell (default shell?), but calling a shell to run a command is the usual way of doing things here. On the other hand, explicitely calling a shell (or entering “shell mode”) would be good to have access to shell environment and variables. It would allow to write commands according to the shell languace (such as csh or zsh commands).
“Edit: And of course osnews doesn’t help me out and it’s all flat, just pretend it’s indented if you see “->”.”
As I mentioned – or complained – before, it would be great to have the <pre></pre> HTML tags enabled in order to post source code or shell fragments in an appealing way.
Edited 2007-04-17 22:24
I’m not so sure things like jam or scons are any better at all this.
Jam surely isn’t much faster, cf Haiku’s build system.
makefiles are not specific to C and C++, and conversely you don’t have to use makefiles to write C/C++ programs.
However, automatically rebuilding things based on dependencies is a problem common to many languages.
There are some nice build systems nowadays. Check out jam, ant, cmake, scons, qmake for instance.
I’m using cmake in my current personal project and I’m very happy with it (although unlike some other recent build systems, on unix platforms it doesn’t replace make and generate makefiles instead – but as I don’t have to modify or even to look at them, I don’t care)
cmake files are pretty straightforward and easy to write, and even complicated dependencies (like having an interface description language to C++ compilation program being built as part of the project and subsequently used to build the rest) are easy to describe.
I am a software developer and a long-time computer user. I started with DOS 3.3. I’ve used Linux since the early Slackware days. The bottom line is some people just absolutely crave these levels of complexity.
I remember my early days of using Linux. I could not for the life of me get dialup networking working. Under Windows 95, it was trivial. You just entered the phone-number, selected the modem, and entered your user name and password – you were done. Under Linux I was supposed to edit chat scripts and mess with PPPD, etc. Finally I found the little utility pppsetup. It asked the same questions as above (phone#, port, user name, password) and BOOM, it connected. What a blessing.
I’ve written some very complicated software, with hundreds of classes, etc. I often write frameworks. When I am done, the user (which might be another developer) is left with very simple but powerful methods to use. However, I also provide a way for the power user to get down to the nitty gritty if they want to.
This to me is the principle of Ruby on Rails. BY CONVENTION, make things simple to use. Make the 90% cases simple. Then, also provide low-level ways for the other 10% to work. Instead, people provide low-level ways to do everthing. Then, only the “macho” can figure out how to do anything.
The same was true when I was a Visual C++ developer. VC Developers were proud that it took 150 lines of code to bring up a window. They should have been ashamed. Encapsulate the _normal_ behaviour. Then, and only then, provide ways for the macho dudes to accomplish and tweak everything at the lowest level.
Whew! I’m done ranting.
both kde and gnome had tools that was very similar to the win9x modem dialog early on iirc.
but yes, early ppp from linux was kinda insane from the cli. but at that time everything linux related was kinda insane coming from win9x or similar…
A nice article. I, however, have a quibble with the title. I think its over complicated and doesn’t describe what’s going on. “Kludgy Conventions” is better.
Someone might mistake it for a KDE troll…
I kid, I kid
I’d say that the Unix filesystem layout is the biggest kudge that we are stuck with these days.
Why? Unless you log in and use your computer as root all the time (against common sense) you might as well ignore all those folders in /.
I am by no means an expert on the *nix file system layout but it does seem very complex to me as a novice, and i think that was the point of the article.
Edited 2007-04-17 20:59
It means %#$ all how the real filesystem is laid out. What matters is that the virtual directory structure, the one you are supposed to see as a user, makes sense.
As far as my memory stretches this very moment, OSX is the only OS who got this right.
At least somebody is trying to change this http://www.gobolinux.org/
Of course.. it’s a copy of the layout from OPENSTEP
It means %#$ all how the real filesystem is laid out. What matters is that the virtual directory structure, the one you are supposed to see as a user, makes sense.
As far as my memory stretches this very moment, OSX is the only OS who got this right.
Huh? Yes and no. It makes little difference to a user if the “system” files are laid out in folders called C:\Windows and C:\Program Files, /lib and /usr, or :bang and :blame. If the folders the user can control are laid out in some unintuitive way ($HOME/pr0n for music, $HOME/catalonia for files about nigeria, for example), that is the user’s fault.
Or do you mean the way OS X hides the names of folders whose names appear in /.hidden? In which case, in many people’s opinion, they still got this wrong: users shouldn’t have to see any of the internals of the OS (even to the extent of hiding /Applications, for example), or have to create folders in their $HOME folder).
Maybe if OS X was based on Plan9 this would work properly: Plan9 for example treats the user’s home directory as his /, and links all “bin” folders to the user’s /bin (thus obviating kludges like $PATH). If they were hidden a la OS X, you would simply drag and drop an application onto a Finder window or a hard disk icon, and it would automagically put all the relevant executables in (an invisible) /bin, all the relevant libraries in (an invisible) /lib, etc. Meanwhile the user would find Applications in the Apple/Applications menu or in the Dock, and use Spotlight to find data files.
Edited 2007-04-18 01:23
“users shouldn’t have to see any of the internals of the OS (even to the extent of hiding /Applications, for example), or have to create folders in their $HOME folder). “
This would be the logical consequence of the statement that users do not use the OS, they use applications; or in more abstract terms: they solve tasks including actions and data, or objects.
But today, users feel theirselves as system administrators, too, or, to be more concrete, they are the administrator of their own system. That’s why the OS has to provide means of system management to them. This does not force the OS to publish anything of the file system layout as long as there’s a GUI that abstracts the tasks that are done (installing applications, setting up services, managing user accounts etc.). In my opinion, MacOS X has a very good and consistent GUI solution for these administrative tasks that do not require the user to feel as a system administrator.
Proper system administration requires more knowledge and experience than just clicking around in a word processor. It takes time for an administrator to educate himself. But today, nobody wants to spend any time; the user wants his work get done, instantly and by itself. Hiding file system internals from him would take a “burden” (and if it’s only imaginated) off him.
In the past, differing between adminstrator and user was a valid means of minimizing problems. Today, OS developers need to think different. Keywords: Abstraction, GUI, safe and useful default settings.
I don’t really agree – it’s already edging into work-around territory if developers have to add more complexity to a system (a separate virtual directory structure) simply to abstract away the underlying complexity of the “real” directory structure. That strikes me as a work-around to avoid having to simplify the actual underlying system.
“The idea that you can put an interface layer above the gas refinery, and the foul-smelling problems beneath it will never percolate to the surface is just that — an idea — a seductive one without foundation in observed behavior.” -Jean-Louis Gassée
But look at the gobolinux project.
They are doing the opposite: Instead of introducing another tree structure (the rpm package database), they use the filesystem as package management, and for backwars compatibility they make all those symlinks which provide the usual UNIX structure to the programs. At some later stage, that symlinking can be omitted and a clean(er) filesystem emerges.
To stay at your gas refinery analogon: They change the gas refinery in a way, so that it does not smell bad anymore, and at the same time they emit a perfume ‘eau de gas’ which lets the leakage-finding dogs still do their work.
Yep, that’s pretty much the same point I was making: whenever possible, it’s preferable to simplify / fix the underlying system – rather than trying to protect/shelter users from it. The ideal IMO is to have an underlying system which is well-designed/transparent enough that it doesn’t need to be hidden from end users.
*linux beginner here* What pisses me the most is when I install an application and the package manager for some reason “forgets” to create icons and I have to go hunting for the executable.
For quite some time I let it go – searching for another app that did the same – then I found out about /usr/bin, but unfortunately there are hundreds of files there and the newly installed app, sometimes doesn’t heave a logical and easy to find name.
An applications/ folder would be nice in *nix, along with a system/ folder for all the crap that no regular user understands – and I don’t mean a hack like Gobo Linux, but it seems that no one has the balls to do it.
Edited 2007-04-18 05:48
Why do you think GoboLinux is a hack? What do you have in mind that is not a hack?
GoboLinux and MacOS X both are hack for the filesystem problem: there is not one filesystem, but two!
The (fugly) classic one and the new one.
The not hackish solution would be to change all the Linux/Unix software to use (only) the new FS hierarchy, but that would be like changing the side of the road English people are using: a synchronisation nightmare!
Couldn’t one change some environment variables like %systemroot% or %temp%, or is this stuff hard-coded in the applications?
Actually, it would probably be more like making everyone else change to driving on the correct side of the road! 8D
Actually, gobo and Mac do it opposite. in Mac they are in the fairly obscure directory names but presenting a virtual file structure that the user can see. In Gobo it is the user friendly filesystem that exists, but a hidden virtual file structure of symlinks exists for applications that are not “gobo-aware” to access. If this became standardised all apps would start to use the new filesystem
I myself have struggled with the Linux filesystem, I am never sure where a program should be installed, on my system there seems to be no less than seven install locations: /bin /usr/bin /usr/local/bin /sbin /usr/sbin /usr/local/sbin and /opt. While I see the need to seperate system binaries from appliaction varibales this seems a little to much. I also know I’m not the only one who gets confused, just look at the shebang line of sample perl scripts on the web to see the wide variety of locations that perl is insalled in.
Interestingly, some countries have changed side of the road. If memory serves Denmark did it and the way they did it was over a period of time new road signs were erected but covered up, and then in the space of one hour (in the middle of the night during which it was forbidden to drive) the right-side signs were uncovered and the left-side ones were covered up. Worked fine.
A more fitting analogy though since the use of symlinks enables a middle transitional phase would be metrication as when this happens usually products are labeled in both units for a period of time.
I couldn’t agree more.
“I’d say that the Unix filesystem layout is the biggest kudge that we are stuck with these days.”
I would not say so, because the file system layout of UNIX (to mention BSD here, but it’s true for Solaris as well) is well intended and serves certain purposes. In Linux, it’s a bit different, as far as I know.
I may give an example from FreeBSD: Everything down from / belongs to the system itself, except of (1) mountpoints, who reside beneath /media (at least as a KDE manner), (2) /home which holds all user data and (3) /usr/local and /usr/X11R6, which contain installed applications that do not belong to the base system. Some time ago, I read /usr/X11R6 will be obsoleted, so you could say, everything installed in /usr/local is a local application.
The user usually does not need to know anything about the file system layout. His files are in ~ and he cannot write anywhere else (except /tmp), so it does not matter.
If you want to know more, just read “man hier” from the good FreeBSD system documentation at http://www.freebsd.org/cgi/man.cgi?query=hier&sektion=7 which contains a very good explaination where everything is located and why it is located there.
MacOS X has done good work to abstract similar structures in a way that they are not obvious at all. You’ll need some “elitist” or “geek” tool to see the real path names.
I hope I did the correct interpretation of what you meant with “Unix filesystem layout”, if not, please correct me.
Yes, this is what I meant. Some interesting info in the link that you posted.
In all fairness, the subj: of my comment was Linux on the consumer desktop.
“In all fairness, the subj: of my comment was Linux on the consumer desktop.”
Let me leave a few words on this: Allthough Linux cannot compete with the tidyness of BSD’s file system layout, it does not matter to the customer. In the usual case the user won’t get any contact to the underlying file system. He creates his own hierarchy in his $HOME which does not affect the system or anyone else. If he is not able to organize his files – no fault of the Linux file system. Regarding application install and update: The recommended tool for this task (that comes with the distribution and is the default tool for package management) should deal with installation, deinstallation and update. It has to know about the file system (where to put which files, which to delete), but that’s no problem because the developers of the default package management tool know their distribution well enough. And adequate abstraction will enable other tools (maybe from other distributions) to do the job better (or at least different), if needed. So, to be polite, the file system layout is not interesting to the desktop consumer. No need for Joe Q. Sixpack or Jane Average to know. They just go there and clickityclick.
> tidyness of BSD
Tidyness? Uh, that’s not a word that I would use for a FS hierarchy which use stupid names such as ‘etc’, ‘usr’ (which doesn’t even mean user!)..
>So, to be polite, the file system layout is not interesting to the desktop consumer.
I disagree. The FS layout should make sense to the user, otherwise you have a brittle system where any ‘advanced’ command looks like magic to the user (a bit like the registry on windows).
“Tidyness? Uh, that’s not a word that I would use for a FS hierarchy which use stupid names such as ‘etc’, ‘usr’ (which doesn’t even mean user!)..”
Allthoug it is mostly pronounced “user”, /usr stands for “UNIX system ressources”, should it le /lsr in Linux, /ssr in Solaris, then?
We had some transparencies in computer science where the professor tried to explain hierarchies: /usr/students/hans and /usr/teachers/fritz.
Historically, /etc did not just contain configuration files, it also held some system utilities, such as /etc/rmt, /etc/mount or /etc/dcheck. It would be understandable to rename it to something like /conf because it serves this purpose today.
You’re complaining about naming conventions, that’s okay, but it has nothing to do with tidyness.
“I disagree. The FS layout should make sense to the user, otherwise you have a brittle system where any ‘advanced’ command looks like magic to the user (a bit like the registry on windows).”
And I disagree, too. The user (NB: user!) should not get in contact with any aspect of the file system (such as it’s usually done in Mac OS X).
Commands in UNIX (and Linux, too) are no magic. You just need – like everywhere – the proper education to know about the relations and the usage, and man is always your friend. Furthermore, users should not be bothered with any commands. They have their GUI programs (or even GUI wrappers for commands), and I may say, they don’t need more.
And what do you consider advanced commands? Most average users are overwhealmed with “dmesg | less” or “camcontrol devlist > /tmp/camcontrol.txt”, but that’s just plain stuff.
While it’s refreshing to see some insights regarding hier, and I agree most people complaining don’t understand the issue at all, I must say that there is some merit to what they are saying.
As I understand it the heir is mostly about two things
1. Indexing known classes of system components
2. Ease manual/automatic system administration
Regarding 1 I don’t really see how a hierarchical organization is superior to, say a relational organization. In any case it could be much improved by actually enforced schemas and types.
Regarding 2 I don’t think there is a case to be made of manual administration anymore. Managing memory should be completely automatic by now.
“As I understand it the heir is mostly about two things
1. Indexing known classes of system components
2. Ease manual/automatic system administration
Regarding 1 I don’t really see how a hierarchical organization is superior to, say a relational organization. In any case it could be much improved by actually enforced schemas and types.”
A file system would have to be based upon a relational database, storing attributes and relations (such as “requires to run” or “does belong to”). A part of them is already represented in the user:group and the rwx etc. attributes used in the usual Linux file systems.
People tend to think in hierarchies rather than in relations. That’s how cognition works. And call me impolite, but… I think a relational file system (and on top of it, relation based OSes and applications) would be to complicated for most average users…
“Regarding 2 I don’t think there is a case to be made of manual administration anymore. Managing memory should be completely automatic by now.”
If you’re talking about RAM while using the term “managing memory”, this is how it works in Linux today. “Automatically” means “done by the OS’s memory manager”.
Manual administration still is a big point, even today in our oh joy oh glory GUI time. If everything fails and you’re sitting on a console with a blinking _, you are glad to have the knowledge and experience to get the thing working again without reinstalling everything. Those of us who are smart enough are even faster doing things “the old way” (tradidional, manual) than anyone else clicking around on colourful pictures with squeaking buttons and dancing elephants.
I think a relational file system (and on top of it, relation based OSes and applications) would be to complicated for most average users…
Well, I wasn’t thinking about files. The whole concept of files is just broken. And I wasn’t thinking about users, why the hell would you expose the internals of the system to users?
If you’re talking about RAM while using the term “managing memory”, this is how it works in Linux today. “Automatically” means “done by the OS’s memory manager”.
Actually I meant managing how, and where to store data on persistent memory. Like harddrives. Some clever design for data management and integration with LVM or similar should be able to handle most cases just fine PnP.
If everything fails and you’re sitting on a console with a blinking _
Well, as a fellow tech geek I have some sympathy with the joy of DIY. But seriously, the correct action is to just take the POS back to the store.
“Well, I wasn’t thinking about files. The whole concept of files is just broken.”
Ah, I see now. Seems we need data stored in EMACS, eleven megagigabytes always continous stream. But seriosly, files are one of the oldest basal concepts in (applied) computer science. What would be a working alternative?
“And I wasn’t thinking about users, why the hell would you expose the internals of the system to users?”
Never show them what they don’t need essentially, they’ll break it otherwise.
“Actually I meant managing how, and where to store data on persistent memory. Like harddrives. Some clever design for data management and integration with LVM or similar should be able to handle most cases just fine PnP.”
This is what file systems are for. FFS or ext2 are not the only one, ISO9660 and even tar are one. Okay, dump is none… but can be used to store data, too.
“Well, as a fellow tech geek I have some sympathy with the joy of DIY. But seriously, the correct action is to just take the POS back to the store.”
There are scenarios where this is impossible, where the only task existing is “Get it working right now!”
files are one of the oldest basal concepts in (applied) computer science. What would be a working alternative?
Why do you need an alternative? Just use whatever data structures is required for the application. Byte streams may be a working serialization concept, but for in memory representation?
Never show them what they don’t need essentially, they’ll break it otherwise.
It’s not about breaking stuff, it’s just not their concern. System development APIs should be designed for developers, not for users.
There are scenarios where this is impossible, where the only task existing is “Get it working right now!”
Obviously we are thinking of different market segments.
Oh, really? I don’t see any explanation why /usr/src and /usr/obj are there where they are. In fact I don’t see explanation for most of the directories. Some existing explanations are 1) arguable like /bin vs /sbin 2) based on old design like /boot that should be replaced with something similar to Linux initramfs 3) stuff related to recovery that should be replaced by booting from recovery media (CD, Flash stick, network, etc..). For more detailed arguments you can read http://www.gobolinux.org/?page=doc/articles/clueless
First off, single user mode is still important. What if the system you are booting from doesnt have a dvd drive? Gonna be kind of hard to fix now isnt it? Many linux system dont as they are used as nodes or network devices. What if the system doesnt allow booting from a removeable media? Many secure systems either have removeable media ports removed or deactivated via BIOS to prevent users from taking data.
As for the filesystem, I question gobolinux’s security. What happens if you have two apps with the same name? Like java or javac? What happens if someone gets a trojaned command on the system?
Lastly, standards are only important if you want your stuff to work with everyone else’s. Otherwise you can do what gobolinux does and go it alone.
What if several sectors of your main hard drive were erased or went bad? If the system cannot boot from an alternative media you’re screwed. If your system doesn’t allow booting from a removable and you’re a sysadmin recovering the system you should call the person who disabled alternative booting and ask him how to enable it. I’m not even mentioning that on a recovery disk you have the best and the most recent tools for recovery unlike the tools on the system that need the recovery.
As for your gobolinux questions: if you install a package with duplicate name of an executable the package will be installed but none of the executables it provides will be in the PATH. You can still call the executables through long path /Programs/IBM-Java/5.0/bin/javac. You can also use package manager to switch between two packages.
This obviously takes care of trojans. If a trojan package provides traceroute executable, it won’t be available in the PATH since the system provided traceroute blocks it.
With regards to standards: gobolinux is source compatible with all other Linux distributions. Is there are a binary standard?
Edited 2007-04-18 05:45
“I don’t see any explanation why /usr/src and /usr/obj are there where they are.”
Maybe it would me more wise to have the obj system compile dir in /var. /usr/src is well placed, it holds the system’s sources (and nothing else).
“In fact I don’t see explanation for most of the directories.”
Excuse me, maybe this is impolite, but… I don’t think you’re judging in an adequately educated way here. Take some time to read more about this topic.
” Some existing explanations are 1) arguable like /bin vs /sbin […]”
The difference is obvious and can be read in “man hier”: while /bin contains “user utilities”, /sbin contains “system programs and administration utilities” fundamental to both single-user and multi-user system. Content in /sbin belongs to system operations, while /bin has a more subsidiary character.
One may want to have the content of /sbin included in /bin. This would be possible because they are located on the / partition. On the other hand, /usr/sbin and /usr/bin could not be placed there, because the /usr subtree usually resides on another partition than /. (Just an addition, the separate partitions serve a purpose as well, they allow special file system optimization and easy dump & restore.)
Furthermore, if you need to recompile parts of the OS, this separation helps you doing maintenance operations according to the groups represented by the hierarchy. Assume there’s a dangerous bug in some utilities in /bin, you just enter:
# cd /usr/src/bin
# make update
# make install
(just exempli gratia)
and have /bin updated, while /sbin stays untouched.
“[…] 2) based on old design like /boot that should be replaced with something similar to Linux initramfs […]”
No, /boot has nothing to do with an initial RAM file system. /boot contains the boot loader (IPX), the kernel and its loadable modules, nothing else. Maybe you want read more, please refer to the system bootstrapping procedures described in “man boot”, see also the “SEE ALSO” section.
“[…] 3) stuff related to recovery that should be replaced by booting from recovery media (CD, Flash stick, network, etc..).”
As it has been mentioned before, physical access to CD, Flash stick etc. are not available everywhere. An OS should provide some means of system maintenance, diagnostics and repair by itself. If they fail, a live file system is handy. And sometimes you even can’t reboot, so you have to use what you have at hand, provided by the OS.
As I said initially, the file system structure is well intended. It needs some time and self traching to know why it is that way. I would not claim it cannot be improved, but filling everything into one dir on one partition surely is no improvement.
Maybe I’m a bit old fashioned, but I’m quite old, too.
I posted this few minutes before I saw the osnews article
http://ubuntuforums.org/showthread.php?t=411979
I agree!
I as a user could easily do with three basic folders:
/home
/removable_media
/.system
The .system folder would be invisible, so I don’t have to see it, as I usually don’t need to see it.
The /home folder swallows everything user related, and the /removable_media provides the mount points.
Then the system design guys could make a mess out of .system, the users could make a mess out of /home, and everybody would still be happy.
As far as I know, gobolinux tris to do something like this, but they keep symlinks which represent the old UNIX convention folders alive. On the other hand, they patched the Linux kernel so that these symlinks are not shown by ‘ls’ or any file manager.
1. what program these days do not have a way to be told at runtime where to get its config file?
2. way over my head. but these days more and more people use a web client for their mail so its mostly a grognard kind of problem i guess
and as more and more isps with any kind of brains use the same host for both (even if behind the scenes they are running on seperate hardware) more often then not the mail client can just default to using the same host as for imap or pop and just aim it at a different port.
still, its mostly a left over from those days when unix ran on big iron and the mail spool was local. when smtp was the only protocol needed as noone would be grabbing their mail from the spool using a external machine.
then its the flexibility involved as now more and more isps require that any mail sendt via their server must come from their ip range. so when your working with a laptop or other portable device you sometimes have to change what smtp to use as you move from ip range to ip range…
and said isps rarely go with imap for their users. that is unless you fork over more cash. at least thats my experience. but then i guess the complaint comes from a office environment…
3. having to key in key sequences to use conference call? if your working with gsm or similar digital phone networks, thats just silly. i could have sworn that all my gsm phones had the ability to start up one of those by me hitting the in call menu and selecting just that option (and no, non of those have been smart phones). same with isdn phones (one of the improvements that system had compared to pots or whatever its called). ok, so it may be and improvement to move that option out so that you can access it directly. but thats good use of a touch screen. and you dont have to be apple or jobs to come up with that idea. but how often one does that kind of call varies highly from person to person.
to me the complaints in this article appear to be very very narrow in scope.
>1. what program these days do not have a way to be told at runtime where to get its config file?
Sure, but that’s still a hack..
For the default configuration, you shouldn’t have to tell an application where is it’s configuration file.
>to me the complaints in this article appear to be very very narrow in scope.
I disagree: complexity coming from historical baggage is really a pain.. Of course when you start from scratch, most of the time, you die because there is no application for your system (BeOS).
and nor do you have to tell it where to find said config file if you put it in one of two places it expects it to be…
and lets not forget that linux and similar is based on a os design for multiuser mainframes. as in, the locations of the files are there for ease of central management, not ease of portable programs.
hell, if you want to launch a program thats in the same dir thats you have currently open on the cli require you to prefix the filename with ./ . if not it will look for the file in the areas defined in path and nowhere else.
remember, with all configs in /etc or $home its simple to do backups. make a copy of /etc, make a copy of /home, dump the packages installed. then when a system needs to be recovered you wipe it, install the packages again, copy /etc and /home into place and its up.
as for complexity because of historical baggage. if it works, dont fix it
>if you put it in one of two places it expects it to be…
Except of course that the submitter wrote (correctly) that default configuration files should be part of the package and packages should be relocatable, so there is no expected places.
>as for complexity because of historical baggage. if it works, dont fix it
But it doesn’t work correctly! Users suffer because of stupid FS naming conventions, non-relocatable packages, broken dependencies, etc.
You’re just so used to the situation that you fail to see the problems..
This reminds me of C/C++ supporters which thinks that “everything’s fine with these language, programmers just have to be cautious”, and back in the real world, users suffer from security hole and the patch treadmill (with safer language like Ada or D, there would still be security patch of course, but much less often).
Write a shell script that does what you want but uses so much CPU time they will gladly give you cron job access just to get you to quit using the “work around”.
#!/bin/sh
while true ; do
if [ “$(date +%k:%M)” = “05:00” ] ; then
do_something_important ;
fi ;
done ;
Note: The functional part of that script will run for the entire minute of 5:00 a.m. Add a check to make it only run once. For the other 1,439 minutes of the day, it will just peg the CPU at 100% for no good reason.
Edited 2007-04-17 21:46
That really is something that someone who isn’t thinking would do. It would be stupid to implement something that consumes the CPU like that when you only need it for 1 minute a day.
A good example, none the less.. of what NOT to do.
“It would be stupid to implement something that consumes the CPU like that when you only need it for 1 minute a day.”
Isn’t it what USB host adapters do with polling (unlike standard interrupt driven serial ports)?
“A good example, none the less.. of what NOT to do. :-)”
I think we need a HOWNOTTO here.
Isn’t it what USB host adapters do with polling (unlike standard interrupt driven serial ports)?
Probably. DBUS and HAL on Linux also seem to poll, as they know immediately when you’ve inserted a disk. (That’s also what caused the annoying clicking sound on Amiga floppy drives, though you could get a utility to stop that. Whether it changed the polling to interrupts, or somehow just silenced the polling, I don’t know).
I really once progged something ALMOST that stupid
.
I have a Rollei slide projector which has an RS232 interface to the computer.
When I wrote a program which lets me “remote control” the projector by the computer I had a problem: How to waste time as efficient as possible?
I was (and am) no great shakes at timers, especially as I wanted to get my timing correct within 1/10th of a second, and the standard Linux timers allowed only for 1 second granularity.
I started with adding and subtracting 1 to a variable within a loop, but as you can guess, this led to 100% CPU utilisation. I ended up with asking the projector for its status, which led to maximum traffic on the 9600baud RS232, but at least kept my CPU at sane 8%.
Why in the world are compilers still not smart enough to not need you to manage memory. Meaning why do programmers still need to use pointers?
Programmers will now try to defend this totally stupid way of programming. STOP NOW!
Instead of spending 40%+ of your time managing memory and debugging that memory management you could instead be actually, oh my GAWD, writing code that actually has something to do with the actual logic and functionality of program making a better user experience and a LOT less bugs and memory holes.
Yes there are languages out there that do this. But they aren’t as main stream as C/C++.
Well I tend to like pointers. I’d imagine it would be rather difficult to have a compiler interpret the context of memory within logic, at least to that degree.
I like pointers because it allows me to have something that is used deeply within a tree of logic, but should be “protected” from the upper branch logic.
I also like pointers from the point of view that it allows me to build complex, but essential and logical data structures too.
I know they can be complex and confusing, but I also tend to think that this is more a product of in-experience and the code itself, rather than pointers themselves.
A lot of people think that multi-threaded programming is difficult, and it is, if you don’t plan it out thoroughly. It’s easy to get “into it” and start cutting code without thinking first, then only to find yourself confused and feeling like you’re in a deep hole.
I’ve never had any problems with pointers or multi-threaded systems I’ve written, I feel, because it was well planned.
Edited 2007-04-17 23:06
Why are so many people scared of pointers and memory management? For all the projects I’ve ever worked with, memory issues and pointer arithmetic were never issues, ie. code once, works everytime. If you understand what’s happening in the backend, there really is no issue. But then again, I cut my teeth with assembler back in the early 80’s, so you had to know what you actually wanted at that level. Migrating to C and using pointers was a no brainer. Understanding memory layout of C++ VTables takes some time, but you just have to figure out how you’d design polymorphism using plain old C and resizable structures on your own, and **smack hand on forehead** this is how C++ handles VTables.
Having said all that, STL containers are still unpredictable, but you really have to step away from the nitty-gritty to use them.
These days, memory management issues boil down to optimising data to fit in level 1 and level 2 caches, and memory alignments.
If you still have trouble with pointers / memory management, it’s time to learn assembler and design your own memory management library. Then everything will just click.
“Why are so many people scared of pointers and memory management?”
It’s because their use needs education far far ahead of clicking around in a GUI. All this low level stuff needs more time to get knowledge about than most of the preconfigured and predesigned stuff. It’s not that pointers are bad by priniple, they’re just not everyone’s first choice. If you write a simple solution for a simple problem, pointers are fast and easy, while they can cause problem in a very big project, especially if you’ve lost the overview. Furthermore, I think this basic C stuff is considered outdated by today’s developers, only geeks and el33tists do use them for useless things.
“But then again, I cut my teeth with assembler back in the early 80’s, so you had to know what you actually wanted at that level.”
Assember is more basal than pointers, and, of course, it depends very strongly on the hardware you’re using. I remember Macro-MOPS on the R300…
“Migrating to C and using pointers was a no brainer.”
On the other hand, character string operations need more “writing” (typing in source code) than, let’s say, Java.
“If you still have trouble with pointers / memory management, it’s time to learn assembler and design your own memory management library.”
As an educational project, I developed a smalloc() / sfree() (“safe malloc”) with CATCH and THROW macros in C many years ago. Surely, not very usable, but all the little things we program are at least a bit of education, experience, and, maybe, fun. So I would not say pointers or asm are useless “because nobody uses them anymore”.
>Programmers will now try to defend this totally stupid way of programming. STOP NOW!
Ahem, you do realise that in some cases GCs have very annoying drawbacks?
1) they tend to use more memory than programs which manage by hand the memory.
2) some implementations induce ‘pause time’ annoying for users in interactive applications.
3) some GCs implementations can induce memory leak (conservative GCs which mistake reference).
I agree that GCs are the sensible default for many (even most of) the applications, but it’s just stupid to claim that GC should be used 100% of the time..
I code in C++ and I don’t spend any significant amount of time debugging memory or memory management issues. You’re doing it wrong.
I also like applications that don’t spend their time shitting up their heap and rely on the GC to clean it up. It force the OS to swap out memory pages full of garbage for no good reason.
GC has its place for some specific usage, but just using it everywhere for everything is plain stupid.
Reference counting is much simpler and faster and is sufficient for most situations.
Indeed that seems really insane.
In BeOS the loaded images are walkable with get_next_image_info(), the first one being the exe, and it contains the full path to it. I use that trick quite often.
http://www.beunited.org/bebook/The%20Kernel%20Kit/Images.ht…
Also, but I consider this a bug (also because get_next_image_info() could do it) argv[0] is overwritten in BeOS by the full path to the exe, to allow the BApplication object to find its resources.
So at the question where am I, you could just say dirname(argv[0]). But of course that can’t work in Unix because they use argv[0] to stuff ugly things like “-sh” for a login shell.
Hmm anyone ever thought about trying to run lsof|grep something to find the path for the mmap()ed lib ?
That could be more portable (hmmm) than using /proc.
What blows my mind is that we had such cool and elegant systems in the 60’s and 70’s, and yet we ended up with this mess. And now there are people who consider eliminating these workarounds to be patentable ideas.
Good grief.
The entire AJAX industry is one gigantic workaround, from using HTTP for everything, including streaming (because companies typically deployed firewalls that just blocked all but the well known ports, and now we even have separate firewalls for web services!), to using HTML and Javascript in ways man was never meant to know (because browsers don’t support SVG and sockets).
I find this issue most frustrating.
thats security vs usability for you…
I agree. AJAX is trying to badly do what XUP (openxup.org) already does (with much more potential), though XUP is very early in its development, as it’s a very small project. Or something slightly different that I’m working on called HICP, which is slowly wandering towards prototype stage.
As for the Unix file system structure, it amazes me that it’s become so ingrained as “the way” that nobody even seems to notice the complete absurdity of having your storage devices located in a directory within the filesystem within the same storage device.
Something like OS-9 (Microware, not Macintosh) does a much better job, in that everything under / is a device, anything under that is a file if it’s a storage device – /hd0/user/johndoe for example. There is a default drive (mirror of one other drive) which serves as “root”. Non storage devices (like a pipe device) can also have a filesystem – whatever the driver likes to present to the world.
Plan 9 does an even better job, with devices available as services which you can mount anywhere you want in your “filespace” (constructed dynamically, not a fixed directory structure on a storage device), and again presents its contents any way it wants to.
Other oddness abounds. Null modems should never have been conceived of, let alone necessary. Why is the digital display memory, displayed on a digital LCD panel, converted to analog to transmit from one to the other – ever, in the past decade? Why doesn’t “ls” produce a panel of icons that you can click on, open, and move around?
I’m doing what I can, but there is so much mental inertial when it comes to “how things must be because they always were that way” that any truely innovative idea just doesn’t get any support until it’s already done and “oh, that was obvious the whole time”.
I think the linux filesystem is fine. You have to remember, that when people created the Unix filesystem structure, it was common to have the filesystem spread across multiple disks. And it you want to net boot, then this feature is important. Windows was designed to be run from one disk, and doesnt lend itself well to net booting. BEsides, as a user,you dont need to muck around with anything outside of $HOME.
the complaints about the *nix file system comes from desktop users. as in, those that have to admin their own system. these often have to go above $home to get stuff done, or fix something…
so yes, the file system works in the office or on a server. but does it really work at home for a non-techie?
Admin will also gain from optimized system with less hacked layer. Faster development, easier to administer, upgrade, etc.
It is more between techie who fear changes (or to learn new things) and techie who welcome them
Edited 2007-04-18 17:22
The point was that unix like systems generally don’t support relocatable binaries.
The actual layout of linux, BSD, unix, etc are irrelevant.
The point is about an obvious, yet missing feature. The various layouts are workarounds.
Regardless of whether you are happy with the layout of your file system or not I agree with the author that this functionality makes a lot of sense both from the POV of developers and end users.
As far as I know Solaris supports $ORIGIN so an application that is located in $ORIGIN/bin knows its libraries are in $ORIGIN/lib etc.
What I don’t understand is: how hard can it be to add this feature to GNU/Linux and BSD? I’ve been looking for something like this for years but I am still waiting for packages to be relocatable.
I understand the need for SMTP versus a single unified all purposed mail protocol.
Almost half of the mail servers only deliver mail to another server. Dedicated SMTP servers make a lot sense given how internet email is designed to work.
POP and IMAP probably should include capabilities to send mail as well deliver it, but that needs to be factored against the redundancy of two (more likely 3 plus) server protocols just for sending mail.
The philosophy of standards is generally to keep the standard simple and not endlessly come up with different standards to do the same thing. Do one thing and do it well.
Worth noting that MS Exchange uses the single server model. Whether it sux or rules depends a bit on who you talk to.
“POP and IMAP probably should include capabilities to send mail as well deliver it”
No they dont and neither to they deliver mail,they only give you access to your mail storage. All mail delivery is done by SMTP. There is in fact not even possible to make POP3 send mail due to it’s design. IMAP could do it but seriously, how much of a hassle is it to fill in one more field in the mail client? IMAP’s insanely complicated enough already.
Also, clients aren’t supposed to use SMTP anymore but mail submission but that hasnt really gained any traction.
JDB’s Internet Mail 2000 is also worth looking into for those interested in alternatives that hasn’t gotten anywhere yet.
http://en.wikipedia.org/wiki/Internet_Mail_2000
I fail to see what SMTP/POP/IMAP has to do with workarounds.
Edited 2007-04-18 07:20
Strangely this article reminds me of the short story “The Lottery” by Shirley Jackson.
It is possible for a program to find out it’s location under Unix. For example, in a shell script you can use “dirname $0”