“Starting today, I plan on posting a monthly vulnerability scorecard for common server and workstation Operating System products. I’m going to keep these scorecards pretty clean of discussion, but you can review my methodology, sources and assumptions.” Note that these results speak only of fixed vulnerabilities; the author aims to include information on non-fixed problems and the time it takes to fix problems as well. You should also read this, by the way.
… from these numbers.
crackers deserve freedom too?
However, he is not counting fixed vulnerabilities for Linux. And his number for XP is ridicilous low compared with the number of fixed vulnerabilities updated through Windows Update.
He is comparing apples with oranges, and he knows it. Explains he is “Exactly how biased am I?” article. The answer is irrelevant, because the real question is: “How flawed is my methodology?” – and the answer is “Critically Flawed”
Yeah! Look! Vista has a perfect score! Guess it’s time to wipe off that FreeBSD partition and also sell those Xserves… Heh.
And it’s absolutely hilarious how he didn’t include pre-SP2 XP. Talk about… bias.
Edited 2007-03-16 18:07
Indeed, the Vista score is a bit misleading to say the least. Vista wasn’t released to the general public at all during the period he is examining. And no competent business will have deployed Vista anywhere but in testing in that period either. So it is quite natural that it has had no fixes.
However, it is not biased to not include pre-SP2 XP. SP2 has been out for years, and everyone at all concerned with security should be running it by now. Just like he didn’t include old version of Linux in his comparison.
Then again, these numbers don’t mean much if you keep your systems up to date. I will be very interested to see his data on non-fixed problems and time to fix. Much more relevant to determining security than fixed issues.
Another addition that would help the credibility of this piece is a detailed view where the vulnerabilities are listed, broken down by component.
Edited 2007-03-16 18:23
Yeah, sure is biased!
Why isn’t my Red Hat 9 install included on there?? Damn you bias Microsoft fanboi.
But some of you should RTFA. Those charts show how many vulnerabilities were fixed. The fact that Vista hasn’t received any fixes (a fact that I sincerely doubt, no matter how good it is, it can’t be perfect) doesn’t mean it doesn’t have any vulnerabilities.
It’s obvious that FOSS software will have more fixes, after all, that code is reviewed by thousands of coders around the world and, hoppefully, those vulns will be fixed before anyone exploits them.
Edit: A typo.
Edited 2007-03-16 18:16
Right he does, and he is thereby deceptive. Otherwise, why call them “Vulnerability charts”? Having a vulnerability FIXED is no longer a vulnerability. He should also indicate how many vulnerabilities have been publicized.
Those charts show how many vulnerabilities were fixed. The fact that Vista hasn’t received any fixes (a fact that I sincerely doubt, no matter how good it is, it can’t be perfect) doesn’t mean it doesn’t have any vulnerabilities.
I agree but it prooves one thing, it’s that Linux has serious security holes despite what are saying linux zealots.
Yeah ? Ok, can we have an example please ?
Just watch the charts dude, can’t you see the red color ?
I agree but it prooves one thing, it’s that Linux has serious security holes despite what are saying linux zealots.
Since your only purpose with that post is to offend GNU/Linux users, I shouldn’t give you any answer. But I’ll try anyway:
1 – It has been already stated that pretty much any GNU/Linux distro includes hundreds of applications and utilities, ranging from simple CD-Audio riiping tools to webservers. Comparing that to an operating system wich includes pretty much nothing is unfair.
2 – Show me ONE source where an objective, common sense-ready GNU/Linux user states that GNU/Linux doesn’t have ANY security holes and I’ll give you (some) reason.
3 – Every distro uses software in different development stages: Some of them include more bleeding edge software (which usually has more bugs) and some of them only include well-tested, patched apps. Not-so-surprisingly, the all-time most secure GNU/Linux distro wasn’t included in the review.
You, sir, aren’t any better than any “Linux zealot”.
Edit: Yes, my grammar sucks.
Edited 2007-03-16 18:39
“I agree but it prooves one thing, it’s that Linux has serious security holes despite what are saying linux zealots.”
That’s a thing I would not disagree, but:
(1) The author compares “fixed vulnerabilites”. If a vulnerability is fixed, it does not exist anymore. So he’s counting things that do not exist. (So your statement should be in past tense: “Linux had serious security holes”.
(2) Fixing vulnerabilities show how good / fast programmers work. Assuming this, the manufacturers of “Vista” hardly do anything, they don’t care anyway.
(3) As it has mentioned before, software included with the OSes (or installed upon them) are interesting, too.
(4) The source contains the vulnerabilites published by the manufacturers itself.
(5) The source contains only the vulnerabilites known, not the vulnerabilities existing in fact.
My judgement: The article is interesting, but says nothing.
And, as you might know from reality, the biggest vulnerability resides between keyboard and chair.
“””
And, as you might know from reality, the biggest vulnerability resides between keyboard and chair.
“””
The engineer in me makes me want to say that we should eliminate that component, then.
“”””
And, as you might know from reality, the biggest vulnerability resides between keyboard and chair.
“””
The engineer in me makes me want to say that we should eliminate that component, then. ;-)”
Well, the engineer in me suggests, we’d actually have to replace that component with one that works better because it’s better educated and has a higher ability of moral judging, but the psychologist in me want’s to give the engineer some sedativa.
I’d like to repeat a thing that someones seem to have forgotten: The article counts the vulnerabilities detected and corrected, so it tells nothing about how secure a system is. The statistics are saying nothing.
I know you were joking, but this links in to a related issue:
Only security freaks, and network managers with inferiority complexes eliminate anything that is a vulnerability. Yes, now that windows has been largely fixed, the user is currently the weakest link in the security chain. But he is also a necessary part of the chain. Too often, the line between security and usability is drawn far too close to security. Features are removed or disabled in software because of ‘security issues’ when the usability/productivity benefits of leaving said features in far outweigh the security drawbacks.
If a vulnerability is fixed, it does not exist anymore.
Yes, I agree, but if there is some fixes, it means there was some vulnerabilities before so it’s quite the same.
And, as you might know from reality, the biggest vulnerability resides between keyboard and chair.
Agreed.
All general-purpose server operating systems have vulnerabilities. OpenBSD proves that even if you obsess about security and only run the TCP/IP stack by default, eventually people will find holes in the TCP/IP stack. It’s inevitable. If you consider vulnerabilities in all of the server packages distributed by the OpenBSD project, the number goes way up. And this is the most paranoid general-purpose server system that a security-minded sysadmin could choose.
This leads to the next point, which is that Windows Server doesn’t come with that many actual servers, whereas most other server platform vendors distribute just about any server software you could want. This figures into any tally of vulnerabilities. Also, as somebody else mentioned, open source systems tend to have more reported vulnerabilities because everything is a white-box attack. Subjecting the code to widespread white-box analysis makes it much higher quality in the long-run, but it also raises the bar for quality because white-box attacks are far easier to craft. In other words, security through obscurity is far from optimal, but it does make the system significantly harder to exploit, and open source systems can’t really take advantage of this.
He does not give you any way (at least that I found) to actually see the vulnerabilities.
I wanted to look at them because that tells you if he falls for the obviously flaw in these kind of graphs.
Which is:
How many pieces of software, or packages are included? For example a typical linux distro includes several mail servers, usually at least two databases, probably a choice of more then one browser, several web programming languages (Perl, PHP, Ruby, Python, etc) and prebuilt apps, etc.
To compare apples vs. apples, this means that with windows you need to include any vulnerabilities fixed in Exchange, MS-SQL, Oracle, Firefox, Adobe products, Cold Fusion, etc.
Very rarely is this done and therefore you are comparing apples to oranges.
I don’t know if he is doing this or not though, and there does not seem to be any way to see the actual vulnerabilities he is graphing.
He does not give you any way (at least that I found) to actually see the vulnerabilities.
How many pieces of software, or packages are included?
Is it that hard to read the teaser? Or the article? Both link to this methodology page with descriptions of which packages are included in the installations used.
The stats are useless. The graph says conveys no information at all, and his explanation on his bias is merely a preemptive strikt against constructive criticism.
Fact is that Jeff Jones is NOT counting fixed vulnerabilities. He is counting the number of binary packages updated as a result of a vulnerability. On most binary distributions in Linux, a single solved vulnerability typical means updating all packages linking against the package with said vulnerability. This gives a high number and a different number for different distributions despite have the same packages and having solved the same vulnerabilities.
His methodology is completely flawed and hilarious and must stem from his lack of knowledge of how to count to 3.
The numbers for Windows XP SP2 fits my experience with Windows 2003 Server (around 24 in that period). OTOH Gentoo has only had 5 or 6 fixes in the same period. And that’s because I simply recompile the vulnerable package (or more for that matter).
For Redhat, Ubuntu and possibly Mac OS X Jeff Jones is not counting fixed vulnerabilities but is counting the number of applications directly or indirectly hit by the vulnerabilities. For Windows he is however counting number of fixed vulnerabilities instead of fixed packages.
He is comparing apples with oranges as is often the case with weird graphs.
“””
On most binary distributions in Linux, a single solved vulnerability typical means updating all packages linking against the package with said vulnerability.
“””
Sorry, but that is not true.
When, for example, glibc is updated, you don’t have to update all the packages that link against it.
But there are plenty of other reasons that his “vulnerability scorecard” is of questionable validity.
When, for example, glibc is updated, you don’t have to update all the packages that link against it.
That’s correct but tell it to RPM-package maintainers
– at least this was a major issue back when I used Fedora, and was one of several reasons for me to switch (switching to LFS was perhaps a bit too dramatic though, but I wanted to learn and be in control, and I was quite frankly pissed 
).
I’d like to hear the other reasons for his “vulnerability” scoreboard to be questionable. What did I miss?
I’ve already posted the stuff that I thought was significant.
Others have done better.
Yeah, Fedora does have a rather nasty case of update diarrhea.
FUD all the way. And by the way, if you want to build a security scoreboard you need to count freebsd in
This was my post on that site about the BSD’s:
From November, 2006 to March 16, 2007, FreeBSD has only issued 5 security advisories:
FreeBSD-SA-06:24
FreeBSD-SA-06:25
FreeBSD-SA-06:26
FreeBSD-SA-07:01
FreeBSD-SA-07:02
http://www.freebsd.org/security/
And arguably the most secure OS on the planet, OpenBSD, has released 10 security updates during almost the exact same time period (OpenBSD 4.0 was released on November 1, 2006). Here is their errata:
http://openbsd.org/errata40.html
Hi,
But still… In that period you can probably find duzens of security advisories for each linux distributions.
Cheers
And arguably the most secure OS on the planet
The ‘arguably’ most secure OS on the planet developers are always seeking at security holes in their code, that’s why they find some.
You would be surprise by the number of security holes discovered if the openbsd developers was applying the same policy only one day on the freebsd code …
Which is the same as the numbers I have for my gentoo installation
However – you forget one thing. The FreeBSD advisories only handle a minimum of packages compared with advisories from Apple, Microsoft and Redhat. Redhat and Ubuntu count in Firefox vulnerabilities. FreeBSD do not despite the vulnerability being crossplatform and relevant for FreeBSD as well. It would be more correct to compare FreeBSD advisories with advisories for LFS and half of BLFS. It gives the same result btw.
Can we remove the whole article please. This article is flawed and should be treated as FUD by the authors own admission.
I’ve been a Director at Microsoft for a little over four years now, in the security group that works to drive security improvement across the company. For that alone, some may condemn me, so let’s dig into it.
Give us the TRUE facts !
Can we remove the whole article please. This article is flawed and should be treated as FUD by the authors own admission.
Can you PLEASE judge the article on its own merits? I have YET to find a SINGLE shred of a pro-Microsoft bias in this article (I still included the link to that page in the teaser, for completeness). The guy is honest about the shortcomings, and he intends on fixing those as soon as possible.
The fact that Microsoft comes out on top* in these results does NOT automatically mean the results are flawed. You should LOOK at the methodology before passing judgment on something. I know that in the present day internet world it is very uncommon to ask such a feat from readers, but you should try it for once.
* The results appear to be in Microsoft’s favour, but since we do not yet know anything about unfixed vuln., it’s impossible to call these results in favour of anything.
“””Can you PLEASE judge the article on its own merits?”””
Fair enough.
To his credit he does address the disparity in included packages between Windows and Linux. But he does seem to perform a bit of voodoo by claiming that he could just click a few check boxes in the install and magically come up with an apples to apples comparison.
If you read his responses in the blog comments (Yes, it’s a blog!), it becomes apparent that he takes the rather bizarre view that only disclosed vulnerabilities are important. He also implies that most of the disclosed ones end up being fixed ones (and that the amount of time to release a fix is not significant) and so fixed vulnerabilities are all he really needs to take into account in his tallies . (Yes, it’s another simple *tally*!)
Add to that the fact that he is a “Director of Strategy” for Microsoft*, and you have to admit that a reasonable person is well within his rights to start getting a bit suspicious.
*For those who subscribe to the view that MS treats security issues as PR problems rather than as technical problems, that would make him a “Director of PR Strategy”, I suppose.
Edited 2007-03-16 19:39
* The results appear to be in Microsoft’s favour, but since we do not yet know anything about unfixed vuln., it’s impossible to call these results in favour of anything.
Don’t we? Seems like there are places online that track these things, and those can be used to show “unfixed” vulnerabilities. Secunia, eeye, frsirt and others come to mind off the top of my head. Unpatched vulnerabilities are known. Just disregarded.
That makes this analysis rather incomplete. There should be consideration of unpatched issues. Days of Risk. Time to patch. Geez… This is a single metric being thrown out, then titled a “vulnerability” report, when it is really a “patches issued” report.
… Get the facts faster that you can say “My windows box was taken over by some cracker“.
Why the hell does someone from Microsoft do this? That piece of information is enough to take the whole thing with a grain of salt…. or a salt dune for that matter, anyway. :-S
Moot point though. It takes longer to say “My windows box was taken over by some cracker” than it does to actually take over the windows box
One point I’ve not yet seen made about this is that there is an implicit assumption that the sources of the raw data (each vendor’s self report of vulnerabilities) are comparable. Different methodologies by each vendor would render comparisons across vendors meaningless.
At this point I personally consider Secunia’s reporting the benchmark to surpass, and this one flaw (among many) brings it far short.
According to Vista Windows Update history, no critical updates have been installed. I have auto-update enabled.
If I was going to score vulnerabilites, I would use SecurityFocus and CVE (cve.mitre.org) to get vulnerability information. As a system administrator I am more interested in what is not patched or fixed, as opposed to what is. Also by searching in such broad terms gives a skewed result.
For example, using SunSolve’s information shows that the results are not limited to the Solaris operating system. For January 2007 there are 19 vulnerabilities listed, as opposed to the 20+ listed in the graph. At least two of them do not affect Solaris 10 at all! The kcms_configure vulnerability does not affect Solaris 10 because it is not part of Solaris anymore, and the Sun Ray Server Admin GUI only affects installations where Sun Ray Server is used.
I would not use this to measure whether an OS is vulnerable or not. There are far better resources for people who are concerned about security, this is nothing more than Jeff trying to make a name for himself online.
and see how hundreds of OSS a$$holes prove the articles wrong.
Now change the article contents and interchange XP with Ubuntu and see how same OSS a$$holes prove the article right:)
Dude,
I hope you don’t mind me calling you “Dude”, but that *is* the “Real name” that you supply in your user profile.
Why show up with guns blazing like this?
Read over the links. Read the posts.
Come to your own conclusions.
But such a confrontational style, and repeated use of the term “a$$hole”, does not improve your credibility any more than it would help an OSS advocate who used that style and nomenclature.
I’m an OSS advocate. If you happen to be a Microsoft advocate, then you would do best to *advocate*, and not sabotage your own position by name calling and being generally confrontational without prior provocation.
Edited 2007-03-16 22:00
Well OSS zealots are a$$holes.. what can i do?
Have you ever seen any other more intolerant and more biased community than the army of these so called OSS zealots or free-loaders (as i like to call them)?
“””Well OSS zealots are a$$holes.. what can i do?”””
Well, let’s look at that.
Are all OSS users a$$holes? Are all OSS users zealots?
I would answer “no” to both of those questions, but I am asking what your answers are.
I see far too much intolerance everywhere, in every community.
Bias, IMO, is something that is not a bad thing. It is a natural, normal, and nearly unavoidable condition.
You have your biases. I have my biases.
It is denial of bias that can be harmful.
Stop every now and then and consider that no community is of a piece… unless that piece happens to be a tapestry.
It turns out that this guy is a Microsoft Employee.
I’m not saying that he’s biased, but the fact that he didn’t disclose the potential conflict of interest is interesting.
It turns out that this guy is a Microsoft Employee.
I’m not saying that he’s biased, but the fact that he didn’t disclose the potential conflict of interest is interesting.
He did disclose it. OSAlert even linked to the disclosure above:
http://blogs.csoonline.com/exactly_how_biased_am_i
No. He didn’t disclose it in the context of that report. This is an important issue. 99% of people who will hit that report will read just that page, not all the other journal entries in his blog.
Even for people who visit Osnews, The summary has 4 different links, only about 5% of the people who read this story will actually get round to reading the pages behind all those links.
I am at least used to the industry, so when I see a supposedly impartial survey promote Windows for its security, I immediately think ‘What is the affiliation of this guy’. In this case, I hunted around the page and eventually was proved correct by a link buried in the comments. What should have happened is that the first sentence of the report should have identified the author’s link to MS, and then everyone would have ended up in a state of enlightenment. (e16?)
where a senior Microsoft security “guru” recommends that Vista vulnerabilities be considered less serious than XP ones.
Hmmm…
I believe we are now seeing the Microsoft spin machine roll into action, now that OneCare has been demonstrated to be both useless and stupidly implemented.
So now to spur sales of Vista, we get treated to one Microsoft bozo who wants Vista vulnerabilities downgraded and another Microsoft bozo who is spinning the comparative rates of vulnerability between OS’s.
This is a joke.
Somebody step forward and directly call both these gentlemen Microsoft LIARS.
NOBODY at Microsoft who is authorized to talk to the public tells the truth. NOBODY – except maybe that one guy who said OneCare shouldn’t have been released.
And he’ll be on the unemployment line tomorrow, no doubt.
Microsoft employees – and I don’t care WHO they are, or where they come from, whether it’s some OSS organization or not, or WHAT their background is with UNIX or anything else – are LIARS by definition. The biggest liar is Bill Gates himself.
The people at Microsoft make the people at Enron look like nuns.
Read Michael Howard’s blog entry about Vista vulnerabilities… It does not say the words that you put into his mouth. Do you not understand how much the online press distorts all news about Microsoft? And you have to understand who’s doing the Vista vulnerability ratings: MSRC– Microsoft’s Security Response Center… It’s not like they’re asking CERT or someone else to do this. Michael Howard just said that he’s not happy that the mitigations in Vista will not really affect the vulnerability rating though they will affect the ease of a hacker actually making an exploit.
Would you explain to me what your background is in security and what expectations you have of any OS? If you can hack a Windows box, I’d certainly like to learn from you, oh guru.
Why are we even paying attention to such silly ‘vulnerability charts’? There are lies, damn lies and statistics. Need I say anymore? OK, I will.
1. The chart counts fixed vulnerabilities. I think it is safe to assume that any code will have a certain percentage of bugs and vulnerabilities. That is just the nature of code. I think we can all agree to this basic statement, yes?
2. Said vulnerability charts ONLY cover fixed vulnerabilities.
We can deduce several points from point 2.
a. OSS has more bugs/vulnerabilities
b. OSS recognises and fixes more bugs/vulnerabilities
Point a. above can be disputed though. We do not know the *total* amount of known bugs/vulnerabilities, because Microsoft does NOT publically admit them. In fact, Microsoft in the past has told bug/vulnerability researchers NOT to post their findings, at least until they’ve notified Microsoft and given them a suitable period of time to fix the issues. This causes:
c. unknown bugs/vulnerabitilities that have not been publicised, but are known by the blackhats. You can bet your bottom dollar that the blackhats will be taking full advantage of said vulnerabilities between the time they were first found, and the time they are patched. The old adage, ‘the early bird gets the worm’ comes into mind here.
So, d. comes into play:
d. How quick are bugs/vulnerabilities noticed and patched?
I think it is safe to assume that more people work on OSS than Microsoft software. More eyes, means more problems are noticed, which means more bugs/vulnerabilities are fixed. I think it is also safe to assume that the cycle of this process is quite fast. Previous Secunia reports back my assumptions here – OSS patches far quicker than Microsoft. The old adage ‘why put off tomorrow, what you can do today’ also comes to mind here.
That said person works for Microsoft also casts a shadow of doubt over the validity of his claims. Even if we allow for the fact that GNU/Linux is used on 2% of world desktop PCs, and Microsoft Windows variants on 96% of them (leaving 2% for Macs), if we work with ratios, you will find that the total number of ‘owned’ systems on GNU/Linux is far lower per capita than Microsoft Windows (the same applies to OS X I might add, it’s lower as well).
Some will argue that this is because GNU/Linux and OS X have smaller numbers of users, so less blackhats concentrate on them. This is partially true, but not holistically accurate imho. The UNIX system of doing things has been around for a long time now, and has always been used for mission critical applications. This is purely because of security and reliability (and scalability for that matter). Take into account that few GNU/Linux systems run anti virus software…imagine running ANY Microsoft Windows variant without anti virus software, how long would it realistically last on an open network?
I’m not saying UNIX or GNU/Linux are totally safe, they’re not. Read point 1 again. All code has bugs. Period. Security by obscurity is never a good design imho – you’re relying on the fact that you’re leaving a security vulnerability open, and that a blackhat hasn’t discovered it yet (and started abusing it). Better to acknowledge the vulnerability publically, have it known, and have 10,000 eyes looking at it and fixing it in a few hours, than leaving it ‘hidden’ in the hope that it won’t be abused.
Another important factor, one that I think is just as important, if not more important than the code issues myself, is the PEBKAC issue. UNIX and GNU/Linux users are more PC competent, and therefore more cautious, less prone to make errors that endanger their systems. Most of this is because Microsoft Windows, has, over a period of time, been dumbed down to cater for the average ‘idiot user’, of which there are many. This dumbing down, makes the system easier to use, but at the expense of security and reliability imho. You can have one, or the other, not both.
Dave
I think you have to look pretty hard to find remotely exploitable security vulnerabilities in WinXP SP2 or in Vista (I’d be happy if you could point one out to me). Running AV is not strictly necessary on the open internet, and the major form of exploit these days is in fact PEBCAK.
I think platform security these days is given more attention than it deserves. I’m confident that finding holes and insecurities in websites with custom PHP, ASP.NET, or any other dynamic content generation will yield far more fruit. Stop trying to pick on Windows, and try to go after live.com, you’ll get more change from that. (Not to mention ‘live’ anagrams with ‘evil’).
I think the real picture here is the ratio of severe vulnerabilities to medium and low risk vulnerabilities. RHEL comes with a lot more applications than XP SP2 or Vista, so it is expected that they will have more fixed vulnerabilities. XP has almost as many severe vulnerabilities as RHEL and it has only a fraction of the applications. XP also has a much bigger ratio of severe vulnerabilities to lower risk vulnerabilities.
Edited 2007-03-17 14:26
So, I can have 1000 vulnerabilities hanging, if I solve one, I am more secure than the guy next door who has 100 vulnerabilities and solved 10?
Precious work, Jeff, and promising too…
I just submitted this to OSAlert too but I made a rebuttal to this here: http://blog.2blocksaway.com/2007/03/18/monthly-security-scorecard-t…
It takes a few more considerations such as patches solved etc. As a matter of principle I will keep this updatd on a monthly basis too.