“LLVM 2.8 was released last month with the Clang compiler having feature-complete C++ support, enhancements to the DragonEgg GCC plug-in, a near feature-complete alternative to libstdc++, a drop-in system assembler, ARM code-generation improvements, and many other changes. With there being great interest in the Low-Level Virtual Machine, we have conducted a large LLVM-focused compiler comparison at Phoronix of GCC with versions 4.2.1 through 4.6-20101030, GCC 4.5.1 using the DragonEgg 2.8 plug-in, LLVM-GCC with LLVM 2.8 and GCC 4.2, and lastly with Clang on LLVM 2.8.”
especially considering that GCC has what 20, 25 years of development behind it, that a long time to learn a lot of optimization tricks.
The great thing about LLVM is that it provides a truly modern basis for future advancement. Plus, it is really easy to work with. I tried some of the example code of using LLVM to dynamically compile some expressions, and it is so easy (well compared to gcc).
I’m not saying gcc is bad, its a great compiler, its what I use everyday, but from what I’ve seen of the source, its certainly not a codebase that I would like to use / extend.
I certainly agree that LLVM’s documentation for developing is superior that of GCC from what I’ve seen/heard. This may be a result of most GCC developers being paid by companies to work full-time on the project and that these companies aren’t that interested in ‘community’ support and thus rather want to focus their efforts on things that improve their bottom line, which judging by the focus of the last releases seems to be code performance.
The plugin-system present in both compilers makes it alot easier to manipulate their respective inner workings but I believe that even here there’s a big difference between the systems in terms of documentation available (in favour of LLVM). Not so surprising perhaps since the GCC project have been wary of GCC becoming a frontend/backend to proprietary solutions. Which I personally can understand and relate to.
Still, GCC is also being improved on in terms of codebase modularity (part of which was the switch to allow c++ code for the compiler suite) so it looks good on both sides. GCC could improve their codebase/documentation/error reporting/compilation speed and LLVM can improve on it’s optimizations.
Either way it’s awesome having these two great free compilers to choose/mix from/with.
A bit confused about what Thom meant with LLVM focused here since the benchmark is of GCC 4.2.1, 4.3.0, 4.4.0, 4.5.0, 4.6 snapshot and Clang 2.8, LLVM/GCC 2.8, DragonEgg/GCC 2.8.
The text on the news on the sidebar are copied word for word by their sources. This was written by the guys at Phoronix.
Ahh my bad, Thom is blameless (I feel a disturbance in the force
)
Phoronix still make mistakes in the article and misinterpret some result (inversing lower is better and higher is better in the comments and things like that), but at least, for the data part. It’s mostly fixed. He now show the range for tests and make them on a wide variety of machines, at least for this one.
Actually there is at least 1 stupid thing again this time in Phoronix benchmarks: that they used the x.y.0 versions of GCC and not the current bugfix versions, e.g. 4.4.5 instead of 4.4.0. Lots of performance regressions get fixed all the time, so it makes a real difference, and right now if you’re using gcc 4.4 you’re certainly not using 4.4.0 (at least you shouldn’t).
Or maybe you just didn’t notice that some charts are of time and others are of speed?
Admittedly they could make that more obvious. They should have all the time-taken graphs right-to-left so that if you look at the end of the bars, the rightmost one is always the best.
It’s a pretty fair description – the text of the Phoronix article is almost entirely concerned with how the LLVM variants compare against GCC, not with GCC itself. I don’t know why they bothered to compare so many GCC versions…
Probably to avoid criticism like “you shouldn’t compare against 4.6, it’s not even released yet”, “llvm-gcc uses gcc 4.2, why didn’t you compare against that gcc version?”, “gcc 4.6 is going to be soo much better, why didn’t you compare against that?”, “gcc 4.3.0 is superior to any other version”, etc, etc.
The credibility of Phoronix of a whole is questionable. The credibility of Phoronix’s benchmarks is questionable, also.
As I said above, this one is a real improvement over previous one, even if work still need to be done.
He now have automated benchmark across multiple systems with multiple version. It seem to run many time, because the lowest and highest score for each test is now displayed. Normally, those are the main criteria for valid results. The whole benchmark source code is also available, so, in theory, every one could replicate it.
It now follow the basic “scientific path” (or whatever it is called in English). I am not saying that it is now perfect, but at least he improved his method since one year, probably because he now sell and support his bench suit. Saying “it’s bad because it come from Phoronix” is just trolling now, it’s not like he ignored all requests, including mine from previous OSAlert/phoronix.com forum posts.
Citation needed please..
Try reading it for awhile. It is the most sensationalist and inaccurate news site I know of. They regularly make up stories (e.g. Steam is coming to Linux).
EDIT: To give you an idea, Phoronix just published a two paragraph article about a 10 word Twitter post.
The benchmarks are “credible”, but if you read the comments you will notice they that rarely have the right compiler flags, settings, versions, etc.
Edited 2010-11-09 22:50 UTC
Well, I comment about that myself, but they do use the same compiler flags for all the compilers so from a compiler by compiler basis the tests are valid.
Not necessarily. For example, -O2 might enable different optimizations for GCC and Clang. -O3 is really the only fair comparison because it is supposed to be the very best optimization that the compiler can do (which is what you’re trying to test, anyway).
While I can agree with that, there’s no reason to assume that it’s not tested with -O3. Actually, unless he has gone in and changed the makefiles I know that atleast some of those packs come with -O3 by default. However, if you have doubts, the best thing you can do (and I’d advocate this is always the best choice if you are interested in compiler performance like I am) is to run your own benchmarks and make your own conclusions.
From my tests (-O3) it’s clear that GCC generally optimizes better than Clang/LLVM, sometimes very little, sometimes quite alot. The few occurences where I’ve had Clang/LLVM beat GCC has been on some of the micro-benchmarks from the language shootout, but I did those tests long time ago (LLVM-GCC 2.6 iirc) so I don’t know if that still holds true.
But I must also say that alot of the bigger stuff I want to test simply won’t compile under Clang as of yet and there is a chance that some of those will perform better than GCC when Clang can eventually build them.
You are very right, and it is easy to at least one big problem in these comparisons: binary sizes are not mentioned. For example, if in one case GCC/LLVM expands more inline functions than the other, we will (probably) see a faster executable, but:
– it doesn’t mean anything about the “quality” of the compiler.
– it really depends only on the heuristics used to decide weather to inline or not.
– the faster executable will be obviously bigger in size.
A much better benchmark of C/C++ programs should consider this, and should *disable* inlining, or consider executable sizes too. Otherwise, the benchmark is useless, it only tells you that different default configurations produces these executables we saw, but we have no idea of why one is faster than the other.
I’ll go on a limb and guess you are a *BSD user?
This was a test of executable performance, not executable size.
You must be joking, seriously. Optimization is a quality of a compiler. One of many, granted, but still undisputable.
Yes, which is an important optimization decision made by the compiler.
Then make such a benchmark, this was again about the fastest performing executable. Not the ‘fastest performing executable’ without inlining. Or ‘fastest performing executable with the smallest size’.
Lol, seriously, ‘useless’??? You are saying that code speed is useless? The compilers where told to create the ‘fastest code’, the benchmarks show the results of that test. It’s not hard to figure that the results were not to your liking but that is neither here nor there. If you want a test about code size vs speed, or just code size or whatever else then go ahead and make one.
As for why one is faster than the other it’s simple, it’s optimization framework does a better job. Why it does that is something the compiler developers will dig into.
I’ve just been trying out ldc, the LLVM D compiler. Looks really promising – good to see that they’re supporting D!
I don’t think it’s really the LLVM team that supports D. LLVM is a compiler framework – anyone can use it to implement a compiler for their favourite language. LDC seems to be hosted at http://www.dsource.org with the source code repo hosted on bitbucket.org and mailing list on Google Groups.
Yes I doubt this aswell, especially since the languages added sofar has been in line with Apple’s wishes (c > ObjC > C++) and I’ve seen no mention of Apple being interested in D. Add to this that Walter Bright of Digital Mars is currently trying to get D into GCC then I’m pretty they did it for LLVM aswell. Anyone with the facts on this? I’ve personally never touched D but I hear good things about it (I’m just too old to learn a another programming language
)
Also, don’t forget the distinction between LLVM and Clang. The backend is being used for everything from Lisp and Smalltalk to assembler. Clang is a C family front end. Even if a D front end had more of an official status, it would probably still not be part of Clang, but as a separate component.