Submitted by On Full Disclosure, there’s a negative analysis of Jeff Jones’ six-month vulnerability report. “Conclusions that are drawn are built on a lack of understanding by the Microsoft researcher. I highly encourage him to go back and take another look, and pare down the results to essential information that is absolutely critical to the conclusions, rather than just ‘Other OS’s have more bugs, see, look at my graphs’.”
Jeff Jones is not stupid. Do you honestly believe that at absolutely no point during his research he had anywhere in his mind the possibility that the severity of the vunrulbilities would be an important factor in writing a balanced report? If he honestly wrote this flawed report unaware of its failings, at any point, that would put him in the same skill level as a ten year old.
The fact of the matter is that this is another shot from Microsoft across the bow of Linux and alternatives attempting to dumb down the truth to a ten year old level, hoping the CEOs will stick to their platform entirely through fear, uncertainty and doubt.
sadly it’s fear, uncertainty and doubt that sells.
Uh, no? Uncertainty and doubt absolutely do not sell anything. If someone is uncertain or doubts a product, they do not buy it. Microsoft are spreading FUD about Linux, not trying to use it to sell their own product.
What Microsoft is trying to achieve is to prevent people buying alternatives. They don’t really have to “sell” their product, many companies simply have no choice in OS or Office Suite because of quickly made decisions based on the least amount of money spent in the short term.
Do you honestly believe that at absolutely no point during his research he had anywhere in his mind the possibility that the severity of the vunrulbilities would be an important factor in writing a balanced report?
Do you honestly believe that at absolutely no point during his research he had anywhere in his mind the possibility that a report favourable to Microsoft would be the most important factor in the presence of non-zero digits in his paycheck?
I wouldn’t mind a large number of zero digits in my paycheck, as long as they were preceeded by a 1.
Edit: Pro / pre lol
Edited 2007-06-29 18:59
Or a 999!
I highly encourage him to go back and take another look, and pare down the results to essential information that is absolutely critical to the conclusions, rather than just ‘Other OS’s have more bugs, see, look at my graphs’.
I don’t think that’s what Mr. Jones was trying to say at all. From hearing the presentation he gave, it seems like he’s more interested in seeing what the fact of the matter is, instead of listening to partisans for any one operating system.
I don’t think that’s what Mr. Jones was trying to say at all. From hearing the presentation he gave, it seems like he’s more interested in seeing what the fact of the matter is, instead of listening to partisans for any one operating system.
Give me a break. If you actually read the report and still think the way you do about his conclusions then you are either an MS shill or completely lacking understanding of security. First of all Jones works for MS so if you don’t think he’s partisan then you’re just naive. Second, counting vulnerability reports isn’t a meaningful metric for security. Third, he wasn’t very clear on what programs he included in the Linux vulnerability count. He was only clear that he didn’t include dev tools, openoffice, or gimp. That leaves a ton of potential software that he counted that has no MS equivalent.
I don’t think this short write up debunked Jones’ report at all. Not that it can’t be debunked but that this little blurb fell far short of doing so.
Jones wrote a report in which he counted vulnerabilities. He told you how he counted them so that you could go back and do the same to see if he counted correctly based on the way he said he did so. Did he make any errors in the count based on the methodology he described?
Then he makes some conclusions about what the counts mean. Are his conclusions reasonable? Why or why not? If you think not, what method would you use to assess security? Why is your methodology superior to Jones’? Describe it fully so it can be checked just like Jones’. What conclusions do you draw using your method?
That’s what’s required to debunk Jones’ study.
With regard to a specific issue, I would consider it legitimate to count vulns in Firefox on RHEL or ubuntu if vulns in IE were also counted. Ditto for any other programs in a linux distro that have a equivalent in a base Windows install (edit vs nano for example). I would also think it would be correct to include driver vulns. For example, if a driver comes with linux, say a network card driver, then vulns in the equivalent windows driver should be considered even if MS didn’t write the driver. The comparison should be based on equivalent functionality.
That’s the direction I think Jones says he’s headed, but I don’t think he quite gets there.
“Then he makes some conclusions about what the counts mean. Are his conclusions reasonable? Why or why not? If you think not, what method would you use to assess security? Why is your methodology superior to Jones’? Describe it fully so it can be checked just like Jones’. What conclusions do you draw using your method?
That’s what’s required to debunk Jones’ study.”
With all respect no. If you find an error in a theory or model, it’s an error, no matter if you have a different and better theory or not.
For one thing IE is tied (supposedly) to the Windows <blank> core. So to lump in an optional browser of a Linux distro and compare it to something you can’t remove from a Windows installation is plain dumb. MS says IE is part and parcel of Windows. Firefox is not.
A Linux distribution ships a complete workstation/server environment. Windows ships an OS with a few utilities, games, and a handful of basic apps. For any serious Windows installation you will have to install your office applications, antivirus, and any other applications you use. A Linux distribution updates ALL applications it ships with and any others that are included in it’s software repositories. Microsoft only updates their software.
Drivers should be included, but only if they are open source drivers with the exception of a third-party driver that causes a vulnerability on the Linux distro where a code change to Linux is needed. And the same for a Windows driver.
Windows, Linux distributions, BSD distributions and MAC OSX are not equal. Linux, BSD and MAC OSX are closer in architecture than Windows is to anything else. Mr. Jones needs to dig deep and really find out what compromises a *nix distribution and make a truly equivalent comparison to Windows. Besides, Microsoft does not disclose all it’s vulnerabilities where as the FOSS community does.
What do you define as the “Windows Core”?
“What do you define as the “Windows Core”?
The Windows (NT) core is correctly defined as the ntoskrnl.exe image (which contains the executive subsystems and the dispatcher) and the HAL dll’s. Everything else, including the network stack, the WDDM/DXGK stack and GDI/User (even the kernel mode module that houses them) are all implemented as loadable kernel mode (driver) images.
IE isn’t, and never has been, part of the core OS, regardless of how many Microsoft or 3rd party applications may have dependencies on IE dll’s and functionality. In Windows Vista, even the usermode shell (explorer.exe), common control dialogs and Windows Update have been decoupled from IE.
For a clearer example of how IE is not part of the core OS, perform a Windows Server 2008 Core installation; you will notice there is no Internet Explorer present (though it will run Mozilla Firefox quite well, sans open/save dialogs .
I agree with everything you said. I didn’t actually expect a response from the guy. It’s easy to say “it’s part of the core!!!11” but it’s harder to back that up.
It’s not part of the core, but Microsoft also can’t remove it without redoing some of their bundled applications and breaking third party applications dependent on IE components (shdocvw.dll, mshtml.dll, etc).
The “Windows Core” is Windows installation without any optional applications installed. These items include the Internet Explorer application for example. IE is married to the base OS. My rationale is that IE is part of a Windows base install, but FF is not a necessity. And correct me if I’m wrong, but I do believe WMP is a core component also.
“Tied” and “Included with” aren’t exactly the same thing.
http://www.microsoft-watch.com/content/security/microsoft_is_counti…
Links to some ZDNet blogs on undisclosed vulns too.
Number of vulnerabilities, even when only the critical ones gets counted, are always disputable. One releases numbers, others attach rationales to such numbers. It’s a never ending story.
However, I’ve seen plenty of “numbers” flowing out other houses (Apple, Linux community et al) so I can’t see why Microsoft should not play this game too.
Fact #1: Vista is proving to be more solid than XP. Bugs exist everywhere but this dude is not so vulnerable as XP was.
Fact #2: I don’t like this guy’s attitude. For years you tricked people into numbers then Hermansen say numbers could be misunderstood. Welcome to real world.
Fact #3: Hermansen negative analysis is not debunking those numbers. He goes abstract since he probably thinks his own numbers will not help him. I’m surprised he doesn’t say “it’s FUD”, since “FUD” term is used everytime someone has nothing to say but he/she wants to write he doesn’t agree.
Fact #4: having more bugs disclosed is “the nature of bug reporting in open versus closed source
software”. Wow! For years the nature of OSS was to be able to *FIX* bugs faster and “more eyes looking, more bugs fixed”. Now that this is not happening, we learn the nature of OSS is to have more bugs publicly disclosed (without caring if they were fixed or not by mythological “Community”)
I have to say I don’t like number per se. They don’t explain anything real. I acknowledge that Microsoft might have fixed more bugs than they disclosed and we surely need to account that.
But now acknowledging Vista is proving comparable to other products in terms of security and probably now doing a bit better would be not being attached to reality of things.
I’ve been always sure that to deliver better products you have first to be able to acknowledge reality of things and then prove yourself better.
Fact #4: having more bugs disclosed is “the nature of bug reporting in open versus closed source
software”. Wow! For years the nature of OSS was to be able to *FIX* bugs faster and “more eyes looking, more bugs fixed”. Now that this is not happening, we learn the nature of OSS is to have more bugs publicly disclosed (without caring if they were fixed or not by mythological “Community”)
You’re absolutely missing the point. The true facts are that, while your average Windows user will not send any kind of bug report, the F/OSS community is way more agressive regarding bugs/flaws, even those which don’t pose a threat for the segurity/stability of your system. But the fact that you have less disclosed bugs doesn’t mean that you actually have less bugs than the competition.
“Mythological”? Hahaha, believe what you want. Once you’re done spitting worthless flames, do some research. You might find that the community is a bit more real than you think.
Edited 2007-06-29 16:01 UTC
You’re absolutely missing the point. The true facts are that, while your average Windows user will not send any kind of bug report, the F/OSS community is way more agressive regarding bugs/flaws, even those which don’t pose a threat for the segurity/stability of your system.
It’s also way more aggressive regarding the unacceptability of bugs and flaws (Exhibit A: The Infamous Ubuntu Non-functioning-X11 Incident).
In Windows’ users’ defence, the last time I remember a bug which crashed Adobe Acrobat happening (which is not the only kind of bug that could happen), Windows attempted to send a bug report to Microsoft, not to Adobe. Unless MS have a means to correct that flaw, or provide Adobe with that information (which judging from the content of the “this is not an MS application” message, they didn’t), that’s not much use.
You’re absolutely missing the point. The true facts are that, while your average Windows user will not send any kind of bug report, the F/OSS community is way more agressive regarding bugs/flaws, even those which don’t pose a threat for the segurity/stability of your system. But the fact that you have less disclosed bugs doesn’t mean that you actually have less bugs than the competition.
No, you’re missing the point. It isn’t necessary for people to report bugs to Microsoft anymore. They can be reported to any number of widely available security bug tracking orgs (ntbugtraq.com, secunia.org, etc). Because it isn’t possible to hide security bugs anymore. Security through obscurity doesn’t work, it’s been proven time and time again, so whether or not the bugs are reported to Microsoft is irrelevant. They WILL be found, regardless.
The F/OSS community needs to get a grip on reality and face facts. Vista is way more secure than its predecessor, and a lot of people don’t like that because they think it reduces the value proposition of their preferred OS versus Vista. And, really, the author of this piece needs to avoid the usual character assassination inherent in putting “researcher” in quotes whenever referring to his analysis. It’s childish. Want respect? Provide data to back up your assertions. I don’t see any of that in the author’s “debunking”.
But it doesn’t matter where users (or any person/bussiness/entities for that matter) report the bugs. My point was that the people surrounding Windows doesn’t usually spend a lot of time looking for errors and writing bug reports. As you say, most bugs will be found, and I agree, but that doesn’t change the fact that F/OSS is likely to have more bug reports (including a lot of issues that many would coun’t as real “bugs”) at a given time than most closed source software. And that’s OK, we are talking about two different development systems, each one with its own pros and cons. The problem I (IMHO) see here is that the MS researcher keeps comparing apples to oranges.
Yes, this “debunking” is way too short and looks to me more like a simple rant than a serious analysis. And it’s pretty obvious that Vista is more secure than XP (not like its a great archievement, though), but the problem here is, as I’ve already said, that the MS “research” compares apples to oranges, wanting to show Vista as the most secure OS out there, when the only thing it can prove is that it’s (for the moment) only more secure than XP.
But let’s face it: comparing two different things just by raw numbers is pretty much only valid for sports matches and not, most certainly, for technology.
Edit: typos, as always >:(
Edited 2007-06-30 03:03 UTC
Security through obscurity doesn’t work, it’s been proven time and time again,
That would seem to be disproven by the stats for XP/Vista bugs versus Linux bugs or even versus OSX bugs. Unless having less bugs makes XP/Vista more insecure, somehow.
That would seem to be disproven by the stats for XP/Vista bugs versus Linux bugs or even versus OSX bugs. Unless having less bugs makes XP/Vista more insecure, somehow.
You’re ascribing way too much meaning to the stats for the first 6 months of vulnerability reports. I never suggested that looking at any specific time period proves or disproves security claims. XP had few vulnerabilities when it was first introduced but, over time, lots were reported. So, evaluating statistics is only practically meaningful when you have a sufficiently large sample period (say, a couple years). That said, the security models for XP and Vista are fundamentally different. Under XP, most users run as Admin; under Vista, they run as standard users, which greatly reduces the attack surface of any potential exploit. Additionally, since IE is the primary attack vector for most Windows exploits, MS spent time making it possible for IE to run in an even lower-privileged “secure mode” that will make it very tough for malware to take root. It is THESE reasons that I assert that Vista will have a better security track record than XP, not on stats alone.
Nice to finally see common sense, I do hope hope that the new security model works as well all know the old one didn`t.
The whole meteric of ‘reported’ flaws is bogus, in OS it’s is normal to document every little flaw found in contrast with Microsoft who will only document (and potentially fix?) what they deem to be important.
It is a clash of cultures, where OS isn`t ashamed of bugs and Closed source is hence it is a flawed meteric.
He did a decent report, he expalined his timeframe and the metrics he used. fact is, XP compared to Vista first 6 months. Vista does come out ahead.
now when comparing Vista to OSS/Linux stuff it gets harder to compare. if you disagree with his report; publish your own with the same metrics and timeframes.
this has nothing to do with his paycheck,microsoft etc.
-Nex6
I would have loved to witness the demos of the withdrawn report nevertheless:
http://www.networkworld.com/news/2007/062707-black-hat.html
“He did a decent report, he expalined his timeframe and the metrics he used. fact is, XP compared to Vista first 6 months. Vista does come out ahead. ”
lol no it didn’t it only came out top for *critical* vulnerabilities XP came out top for percentage fixed vulnerabilities…Vista came out last.
<it>this has nothing to do with his paycheck,microsoft etc. </it>
Yeah, sure, the result has nothing to do with the method (do you still believe in Santa Claus?)
Would a “security researcher” paid by Apple/Novell/Red Hat/… have choose the same timeframe / metrics? No? Well, then …
Don’t waste your time on this, it is simply a rant and nothing else. Jeff Jones report had real content. This is just another rant from a stallman baby.
Kristian Hermansen has done a dis-service to OSS in general by coming up with such crap. If he really wants to debunk the report, he should do some research, come up with some data.
Breaking the Legend of Trusted Computing and Vista(BitLocker):
http://www.nvlabs.in/?q=node/32
And why should we pay attention to his rantings? Just do a quick google search of his name. He seems to be some random Ubuntu dev. I didn’t realize that emails from random people on security lists were so authoritative as to debunk anything. If it were some well-known researcher on a low-volume list, I’d see the value in it, but I’d love to be enlightened as to why this guy is important.
Kristian Hermansen is a security researcher and developer at Cisco Systems. He spends most of his time hacking around and breaking things. Recently he has been striving to build a massive computational infrastructure based on virtualizing technologies.
A simple Google search pointed to that info located at http://www.ubuntulive.com/cs/ubuntu/view/e_spkr/3455
The truth lies somewhere between the two extremes of the opinions and numbers bantered around by Jeff Jones and Kristian Hermansen.
It is valid to state that comparing disclosed/reported vulnerabilities is difficult between Open Source solutions and Microsoft’s products. The simple fact that Microsoft doesn’t report “silently” fixed vulnerabilities that result from auditing surrounding code and components, when an issue is reported publicly, really does make the vulnerability counting exercise a pointless one. Hermansen, along with others, is correct in pointing this out and Jones would be better served by using other points of comparison.
WITH THAT SAID, Hermansen veers straight into FUD territory as soon as he started making totally unsubstantiated claims regarding the security of the rewritten network stack and citing Symantec’s already debunked whitepaper. The fact is Vista, for whatever its other flaws might be, is vastly more secure than XP due to far more strenuously audited codebase, better compile-time checking, architectural changes (MIC, Session 0 isolation, service lockdown and SID isolation), implementation changes (ASLR, long-lived pointer obfuscation, x64 patch guard, etc.) and much better user environment defaults (UAC, IE Protected Mode).
Is Vista more secure than other OS’s? That only time will ultimately tell. However, the fundamentals do look good.
It is valid to state that comparing disclosed/reported vulnerabilities is difficult between Open Source solutions and Microsoft’s products. The simple fact that Microsoft doesn’t report “silently” fixed vulnerabilities that result from auditing surrounding code and components, when an issue is reported publicly, really does make the vulnerability counting exercise a pointless one.
This is not completely true. It’s true that Microsoft doesn’t always disclose details about their fixes but they disclose how many of them and how important they are. You just need to count how many updates will be released and how many of them are rated critical (they don’t like about this).
Even when they “silently” patch a bug, they need to release an update and rate it.
(of course it’s true that a single fix could patch many bugs in different places but they can be considered “atomic” as they solve a single problem)
What I’m looking forward to with Vista is DRM-enabled malware. Jolly good time.
Howdy
The first report was more like ‘lets limit the data untill we can get something favorable’ and this response is really to short, would be nice to debunk this in a precise way but there are so many FUD reports comming out these days it would be hard to find the FREE time to debunk them all.
And what is with all the attacking comments, looks like the MS employee/PR machine has taken a scientology point of view and decided to attack anyone who disagrees with their obvious one sided view.
What is the fuss all about? Linux has always had more vulnerabilities, and this is because of the OPEN method of development, many eyes make light work of bugs in the code.
The most important issues are severity of the bugs, and if it’s anything like FireFox vs IE, then it wouldn’t surprise me (IE bugs are, as a trend, more critical, even though FireFox has more bugs). Furthermore, Linux and other open source have a record of patching bugs far more quickly than their proprietary counterparts.
I think you can logically find from all of this:
1. Linux and open source have more bugs because they are developed openly, with the src code freely available
2. Linux and open source code is patched more quickly
3. Proprietary software is always bound to have more hidden bugs and flaws, because it is closed src, and by nature that means less eyes to check the code.
Let’s look at Vista – six months on and still no SP. I believe very few updates have been released for Vista (correct me if I’m wrong, I don’t use, or plan to use that crap).
Dave
As stated by several poster’s, uSoft is comparing apples and oranges. Besides, how does one expect bugs in Vista to rear their ugly heads when your PC has been slowed down so much that they never get a chance to execute? The poor things are dying from old age and neglect! LOL:))
Linux Weekly News has (again) one of the best, most balanced and detailed articles on the subject, related to the original J.Jones study:
http://lwn.net/Articles/239457/