The python final release of Python 2.4 was just released. Python 2.4 is the result of almost 18 month’s worth of work on top of Python 2.3. New features are, but not limited to, function decorators, generator expressions, a number of new module and more.
Python 2.4 Released
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSAlert.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
Yes, I hope somebody dose a benchmark soon. I want to know how much faster it is.
great work from the py team! I am waiting for ruby 1.9 (and the watershed 2.0 release……)
Ya know, I’ve seen this news in a couple of places, and I haven’t seen a single person concerned with whether this release affects performance. I’d really like to know if it does! Previous releases are perfectly suitable for me; the only thing I’d look for in a new release of Python would be performance enhancements (not that the changes in this release aren’t welcome).
“Yes, I hope somebody dose a benchmark soon. I want to know how much faster it is.”
I don’t think performance was a major goal of this release. In my tests (primarily on Windows), Python 2.4 is not faster than Python 2.3, all things considered.
A few specific elements such as list comprehensions have been optimized, and some standard library modules have been altered to use more optimal data structures under the hood (e.g., the Queue module now uses the new ‘collections.deque’ for its internal container rather than ‘list’). Most of these optimizations were implemented by Ramond Hettinger–thanks, Raymond!
Other elements, such as the interface layer between the Python interpreter and C extensions, are considerably slower under some circumstances. The move from MSVC 6 to 7.1 has nearly doubled the size of the Python DLL (953 KB to 1824 KB), which seems likely to negatively impact performance on cache-poor CPUs (though one never knows without testing).
MinGW seems to suffer more overhead in operating on the MSVC 7.1 runtime than it does on the MSVC 6 runtime, so if you’re using MinGW as your C compiler, prepare for a slowdown with Python 2.4. On the other hand, MSVC 7.1 typically generates somewhat faster code for complicated C++ programs than 6.0, so if you’re using C++ heavily and are able to make the transition to MSVC 7.1, you’ll likely see a performance improvement. (By the way, the free “command-line toolkit” version of MSVC 7.1 can be bullied into working for Python 2.4 extensions written in both C and C++, if you’re willing to expend enough effort on setup.)
In my tests, programs written in nearly pure Python are 1-2% faster. Those that depend heavily on C or C++ extensions are at least 5% slower under Python 2.4-MinGW than Python 2.3-MinGW, but somewhat faster with Python 2.4-MSVC 7.1 than Python 2.3-MSVC 6.0.
I am most impressed by the language improvements, regardeless of performance gains. The “set” builtin for example is very good, just to name one.
I just downloaded Python 2.4, updated the cx_Oracle modules for Oracle API hooks, and everything works perfectly.
Can’t wait to start using some of the new features.
To me the ability of the language is more important than the sheer number crunching power speed. What does it matter if its a bit slower, compared to the new features they have added to it… imo…
??? Performance almost always needs to be number 1 on a developer’s list for all but the most trivial of applications. Languages should always attempt performance gains between releases, as should custom applications developed in said language. What matters about it being a bit slower when new features are added is that this can become recursive and hobble the language down the release line, effectively killing off new feature justification.
Before I get flamed, I should mention I am NOT targeting Python w/ my previous remark; this holds true for all languages.
*waits for the pacman repositories to be updated
if people were always worried by performance they would be writing stuff in asm all the time. Luckily most of the applications run just fine with slow languages like python if alghoritms are tuned ok.
Why is speed important? Simple, writing games with Pygame and Python gets a lot more interesting with some extra speed. Besides after the Pie-thon/Parrot vs. Python thing you would expect that SOME type of performance improvement took place.
“””
What’s new in Python 2.4
Here are the (subjective) highlights of what’s new in Python 2.4.
Faster
A number of modules that were added in Python 2.3 (such as sets and heapq) have been recoded in C. In addition, there’s been a number of other speedups to the interpreter. (See section 8.1, Optimizations, of the “What’s New” document for more).
“””
haha
always nice to see ArchLinux users wait for some minutes/hours for binary packages to be released. Tell me Keith did you flag it out of date? Cause I went now and it’s already flagged out of date. Niceeeeeeee for Python and for AL 
All,
I have been writing code for approximately 20 years and have worked with a number of different programming solutions. The evolution of languages like Python have been a blessing in terms of productivity, ease of use, fun, etc. However, the lack of performance is felt with almost every heavy weight python application. Yes I have delivered production level projects utilizing python and it worked just fine for projects that were not performance bound. On the other side of the coin, it was over looked on a number of projects because it did not perform well in testing when compared to other tool sets.
The point is this: Having tools like python allow you to enjoy the benefits mentioned above, but if it can only be used in a handful of projects where performance is not an issue, then it negates many of its good points. I have talked to Guido V. about this and my sentiment was reflected by the leadership of the Chandler project. “Python is great but needs a major performance boost”. The real advantage of a language like Python would be to replace the use of heavy weight tools.
Is performance important? With out a shred of doubt.
Obviously speed is important, no doubt.
However, when hardware is typically gaining in performance around 20% per year – why the fixation on *pure* performance?
Think of it this way; Is a python app running on a 3.2Ghz CPU faster than a similar C application running on a i386 33Mhz computer?
In almost all cirumstances it will be faster than the C app.
Add to that, how long did it take to write the C app versus the python app?
This is not to say that we should embrace bloat, however, a pragmatic approach needs to be taken that acknowledges that most bottlenecks these days are with networking / database access. Python offers a wonderful increase in developer productivity and next year, that same app will automagically run ~20% faster.
Moreover, python offers a very simple way to have the best of both worlds. Re-write the time critical areas in C (or C++) and import it as a module — tools like swig make this an almost no-brainer.
Nick — I’m pretty new to Python, but I’ve read that it’s possible to incrementally replace performance-critical portions of your Python code with compiled C code. Is this a solution, or does it simply lead to your project containing so many modules coded in C that it defeats the purpose of using a high-level language like Python in the first place?
Well, as you said, one nice thing about Python is the ability to replace modules incrementally with compiled C code. If you end up with all C code at the end, that’s fine, Python allows you to write it first, write it quickly, and once you have the algorithm implemented and working, optimize it with C.
johnMG wrote:
“””
… I’ve read that it’s possible to incrementally replace performance-critical portions of your Python code with compiled C code. Is this a solution, or does it simply lead to your project containing so many modules coded in C that it defeats the purpose of using a high-level language like Python in the first place?
“””
I can confirm that this works well most of the time, because the claims you’ve heard that “a program spends 80% of its CPU time executing 15% of its code” are usually true. Python includes good profiling tools, so identifying these performance-critical sections is easy.
The write-in-Python-and-incrementally-recode-in-C model becomes less effective as the complexity of the performance-critical code increases, though. At a certain point, it’s more effective to move an entire subsystem from Python to C than to try to code only the performance-critical sections in C. Making frequent transitions between the C layer and the Python layer can slow things down, both because there are inherent performance penalties in the transition between one language and another, and because executing any Python code–no matter how trivial–in a tight loop causes the Python interpreter to compete with the native code for the CPU’s cache.
So in especially complex scenarios you move a broader swath of the code to C (or C++, if the code is complex enough to warrant the headaches) and make the entry points more “coarse”. Once you start using complicated native subsystems from Python, particularly subsystems that aren’t Python-specific and thus aren’t integrated with the Python garbage collector, keeping track of memory ownership can become quite tedious.
I have considerable experience at writing performance-critical applications in a combination of Python and C/C++, and I can confirm that most of the positive propaganda about the ease of integrating Python with native code is true. Even so, I wish that Python had a Lisp-like ability to morph into a low-level, statically typed, performance-oriented language in those few sections of the code where it’s necessary. It seems to me that this would solve many of the problems that emerge in particularly complex scenarios.
All,
I agree that in many cases throwing hardware at things can resolve the issue. However, there are some issues that still exist with that solution.
1. Budgets & Policy: In all of the production environments that I have worked in, a budget was set for hardware and most organizations squeeze the blood out of every machine that they can. For example, I worked for a large company that servers 10,000,000+ customers in real time. Our team was in the critical path for the customer and our servers provided millions of transactions per day. Without those services all of those customers would be without service. Would you believe that we had a number of critical servers that were built from old workstations that wouldn’t even run windows. Most of them were scrapped by the IT department and had coffee rings on them. My personal workstation was an incredible pII 400
I know that it would be unrealistic to expect python or similar languages to be as fast as c, but we aren’t alway able to upgrade 200+ servers every year. On the other end of the spectrum, just getting a dedicated server in a small location is often a minor miracle. So if we test python vs X on the current hardware and x is 40% faster then it becomes difficult to push python as a primary Dev tool. I have tried 
2. Developer Skill: Now that I own and manage my own company, I look at tools like python as a productivity tool to save on development time, and maintenance costs. If we end up with a bunch of c/c++ code we end up needing a developer that is capable of developing/maintaining that code. As we know python is a much shorter jump than writing GOOD c/c++.
Of course I believe that we should be using tools like Python, Ruby, …, and do on a regular basis, but we can’t ignore performance. I believe that the two are not mutually exclusive, there just hasn’t been a great enough push to make it happen yet. Although projects like IronPython are helping.
A side note: Python, Ruby, …, are perfectly usable for many applications and would encourage people to start using them. The more that people use these tools the better that they will get. The rapid growth of these communities is a testimony to the quality of these tools and there promising future.
–Nick
Why not just use a language that both has very fast implementations and is fun to write, like O’Caml?
The name isn’t good enough to run mission critical apps.
Why not just use another language? Simple, I like Python, and for most it is fast enough. Just for Games its speed isn’t all that great. The OCaml SDL bindings are apparently early in development and the Python bindings are done. Then there’s the thing about the numbers of programmers that use Python and those that use OCaml. The Python community just is larger, and I know where to find help.
However, when hardware is typically gaining in performance around 20% per year – why the fixation on *pure* performance?
Think of it this way; Is a python app running on a 3.2Ghz CPU faster than a similar C application running on a i386 33Mhz computer?
In almost all cirumstances it will be faster than the C app.
Yeah, right. Only that the C apps of today are also run on 3.2Ghz machines. So Python will always be slower than C. Then, as the faster C programs changes people’s expectations of speed, it will also always feel slow.
Add to that, how long did it take to write the C app versus the python app?
If the C app cuts it, but the Python one not, because it is unbearably slower, then it doesn’t matter.
This is not to say that we should embrace bloat, however, a pragmatic approach needs to be taken that acknowledges that most bottlenecks these days are with networking / database access. Python offers a wonderful increase in developer productivity and next year, that same app will automagically run ~20% faster.
Hmm, I don’t think application benchmarks show that computers get 20% each year. Also, the app will have to run in a variety of older machines (2-3 years in the past at least) and not everyone changes their CPU each year.
Moreover, python offers a very simple way to have the best of both worlds. Re-write the time critical areas in C (or C++) and import it as a module — tools like swig make this an almost no-brainer.
Not so simple, and it fcuks up the portability of your program, whether in binary form or not.
What I think python needs is an *easy* way to compile things natively as well as to bytecode, like I’ve just been reading OCaml can. I’m not sure how feasible this is, but it would be great. Python is a great language but it will still be dismissed as a scripting language by many until it an be compiled natively. In the meantime, improvingn the optimising of the bytecode compiler (iirc it only removes asserts at the moment, things like loop unrolling would be useful) would be a good thing as well.
If you want better performance, profile the stuff, and recode the important sections in Pyrex, not C.
Why? Pyrex is almost-python, so you don't have to rewrite, only retouch. And performance is almost-C.
Also, you avoid the tedious memory-management and data-marshaling you have to do to handle C extensions.
And if you have a C lib you want to use, using it from Pyrex is semi-trivial (at least not much harder than from C) making a thin wrapper module for the rest of the program easy to achieve.
For example, I wrote a Pyrex module to expose a large chunk of libmikmod to the python side of the app. It took about two hours.
mikeyd wrote:
What I think python needs is an *easy* way to compile things natively as well as to bytecode, like I’ve just been reading OCaml can. I’m not sure how feasible this is, but it would be great.
I’m not so sure you’d see too much of a speed improvement. Python is such a dynamic language (that is, so much is handled at runtime) that even if you compiled the Python to machine code, it would still have a very large runtime that it would rely heavily on to handle the dynamic aspects of the language. It may be a case of “6 of one, half dozen of the other” — either the compiled Python interpreter does the work, or the compiled runtime.
Roberto’s suggestion to use Pyrex sounds interesting. I’d heard of Pyrex, but thought it was some sort of “hotspot” type JIT compiler. Didn’t realize it involved any “almost-Python” code… Thanks for the heads-up Roberto.
Nick wrote:
So if we test python vs X on the current hardware and x is 40% faster then it becomes difficult to push python as a primary Dev tool. I have tried
Did you have a look at Pyrex?
Pyrex is an almost-python->C compiler.
The JIT compiler you are thinking of is probably Psyco
The JIT compiler you are thinking of is probably Psyco
Heh. Right you are!
Whoops.
I asked Nick:
Did you have a look at Pyrex?
but that question was also directed at Anonymous (IP: —.htcnet.org), and anyone else who’s profiled and used C modules with their Python code in an effort to speed up execution.
What about python modules? Will Pyrex just work with them? Or will I need Pyrex modules instead?
You can call python modules from Pyrex. However, those calls execute at python-speed.
You can mix calls to python code with calls to C code in the pyrex modules. That tends to provide good speed without being too ugly.
The main care you have to take is to use C types for stuff that you can , so you are not constantly (de)marshaling data to the python interpreter.
You do that by declaring variables to be of a specific c type.
You can call pyrex modules from python, too.
But really, just read the Pyrex docs, it’s simple.
Read up on it at http://nz.cosc.canterbury.ac.nz/~greg/python/Pyrex/
It looks like typical usage is to write your program proper in Python, then write some performance-critical modules in Pyrex.
However:
http://www.freenet.org.nz/python/embeddingpyrex/
foljs:
Yeah, right. Only that the C apps of today are also run on 3.2Ghz machines. So Python will always be slower than C. Then, as the faster C programs changes people’s expectations of speed, it will also always feel slow.
You miss the larger point – which is from the begining of time (or about 1990) – people have been complaining that interpreted languages were too slow. However, as hardware has gotten quicker the landscape has changed. I promise you that 99% of the application that you run on a daily basis are not computationally bound. Most are limited by IO, either network or disk.
If the C app cuts it, but the Python one not, because it is unbearably slower, then it doesn’t matter.
Wonderful display of circular logic. There are obvious cases (ie. kernels, device drivers, computationally intensive applications) that will REQUIRE C or C++ – there is no doubt about that. However, lets take the case of a simple web app. Lets say that the C app returns a web-page in .001 seconds and the python takes .1 seconds. When transmitted over the web / redered by a browser – what is the *real* difference between those two? If you said one was 100x times quicker, you would be wrong.
(responding to my claim that python modules can be rewritten in C)
Not so simple, and it fcuks up the portability of your program, whether in binary form or not.
Ahh yes, and you never have to worry about that with pure C programs do you? Doh!
“I don’t think performance was a major goal of this release. In my tests (primarily on Windows), Python 2.4 is not faster than Python 2.3, all things considered.”
It is definately faster in some areas because some of the modules that were written in Python have been replaced with modules that are written in C and compiled to native code.
test_decimal 49.79 -> 45.87 8% improvement
pybench 7542.02 -> 6607.81 12% improvement
parrotbench 53.874 -> 49.141 9% improvement
There are plenty of more important performance gains not measured by the above benchmarks.
For instance, the Queue was rewritten and now only blocks when it is full. Also, the underlying fifo data structure is now collections.deque() instead of list() — this changes an O(n**2) process to O(n) resulting in much better scalability. Similar improvements were made to asyncore and other modules with fifo queues.
The implementation of appends and pops for lists was improved to make many fewer calls to the underlying realloc routine. If your compilere supplied an excellent realloc, you will only see a minimal improvement. For those with a crummy realloc, the improvement was substantial (over 60% in some cases).
List comprehensions now run much faster. Genexps offer comparable performance.
Fundamental building blocks like set, heapq, and bisect have all been re-written in C offering speedups ranging from double to tenfold.
Real programs ought to see an excellent boost in performance (both speed and memory utilization).
P.S. the pystone benchmark was not included above because it is not suited for measuring the relative performance of python implementations; instead, it is a tool for predicting relative performance in different hardware/software environments.