****************************

  What's New in Python 2.4  

****************************

:Author: A.M. Kuchling

.. |release| replace:: 1.02

.. $Id: whatsnew24.tex 54632 2007-03-31 11:59:54Z georg.brandl $

.. Don't write extensive text for new sections; I'll do that.

.. Feel free to add commented-out reminders of things that need

.. to be covered.  --amk

This article explains the new features in Python 2.4.1, released on March 30,

2005.

Python 2.4 is a medium-sized release.  It doesn't introduce as many changes as

the radical Python 2.2, but introduces more features than the conservative 2.3

release.  The most significant new language features are function decorators and

generator expressions; most other changes are to the standard library.

According to the CVS change logs, there were 481 patches applied and 502 bugs

fixed between Python 2.3 and 2.4.  Both figures are likely to be underestimates.

This article doesn't attempt to provide a complete specification of every single

new feature, but instead provides a brief introduction to each feature.  For

full details, you should refer to the documentation for Python 2.4, such as the

Python Library Reference and the Python Reference Manual.  Often you will be

referred to the PEP for a particular new feature for explanations of the

implementation and design rationale.

.. ======================================================================

PEP 218: Built-In Set Objects

=============================

Python 2.3 introduced the :mod:`sets` module.  C implementations of set data

types have now been added to the Python core as two new built-in types,

:func:`set(iterable)` and :func:`frozenset(iterable)`.  They provide high speed

operations for membership testing, for eliminating duplicates from sequences,

and for mathematical operations like unions, intersections, differences, and

symmetric differences. ::

   >>> a = set('abracadabra')              # form a set from a string

   >>> 'z' in a                            # fast membership testing

   False

   >>> a                                   # unique letters in a

   set(['a', 'r', 'b', 'c', 'd'])

   >>> ''.join(a)                          # convert back into a string

   'arbcd'

   >>> b = set('alacazam')                 # form a second set

   >>> a - b                               # letters in a but not in b

   set(['r', 'd', 'b'])

   >>> a | b                               # letters in either a or b

   set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])

   >>> a & b                               # letters in both a and b

   set(['a', 'c'])

   >>> a ^ b                               # letters in a or b but not both

   set(['r', 'd', 'b', 'm', 'z', 'l'])

   >>> a.add('z')                          # add a new element

   >>> a.update('wxy')                     # add multiple new elements

   >>> a

   set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])       

   >>> a.remove('x')                       # take one element out

   >>> a

   set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])       

The :func:`frozenset` type is an immutable version of :func:`set`. Since it is

immutable and hashable, it may be used as a dictionary key or as a member of

another set.

The :mod:`sets` module remains in the standard library, and may be useful if you

wish to subclass the :class:`Set` or :class:`ImmutableSet` classes.  There are

currently no plans to deprecate the module.

.. seealso::

   :pep:`218` - Adding a Built-In Set Object Type

      Originally proposed by Greg Wilson and ultimately implemented by Raymond

      Hettinger.

.. ======================================================================

PEP 237: Unifying Long Integers and Integers

============================================

The lengthy transition process for this PEP, begun in Python 2.2, takes another

step forward in Python 2.4.  In 2.3, certain integer operations that would

behave differently after int/long unification triggered :exc:`FutureWarning`

warnings and returned values limited to 32 or 64 bits (depending on your

platform).  In 2.4, these expressions no longer produce a warning and instead

produce a different result that's usually a long integer.

The problematic expressions are primarily left shifts and lengthy hexadecimal

and octal constants.  For example, ``2 << 32`` results in a warning in 2.3,

evaluating to 0 on 32-bit platforms.  In Python 2.4, this expression now returns

the correct answer, 8589934592.

.. seealso::

   :pep:`237` - Unifying Long Integers and Integers

      Original PEP written by Moshe Zadka and GvR.  The changes for 2.4 were

      implemented by  Kalle Svensson.

.. ======================================================================

PEP 289: Generator Expressions

==============================

The iterator feature introduced in Python 2.2 and the :mod:`itertools` module

make it easier to write programs that loop through large data sets without

having the entire data set in memory at one time.  List comprehensions don't fit

into this picture very well because they produce a Python list object containing

all of the items.  This unavoidably pulls all of the objects into memory, which

can be a problem if your data set is very large.  When trying to write a

functionally-styled program, it would be natural to write something like::

   links = [link for link in get_all_links() if not link.followed]

   for link in links:

       ...

instead of  ::

   for link in get_all_links():

       if link.followed:

           continue

       ...

The first form is more concise and perhaps more readable, but if you're dealing

with a large number of link objects you'd have to write the second form to avoid

having all link objects in memory at the same time.

Generator expressions work similarly to list comprehensions but don't

materialize the entire list; instead they create a generator that will return

elements one by one.  The above example could be written as::

   links = (link for link in get_all_links() if not link.followed)

   for link in links:

       ...

Generator expressions always have to be written inside parentheses, as in the

above example.  The parentheses signalling a function call also count, so if you

want to create an iterator that will be immediately passed to a function you

could write::

   print sum(obj.count for obj in list_all_objects())

Generator expressions differ from list comprehensions in various small ways.

Most notably, the loop variable (*obj* in the above example) is not accessible

outside of the generator expression.  List comprehensions leave the variable

assigned to its last value; future versions of Python will change this, making

list comprehensions match generator expressions in this respect.

.. seealso::

   :pep:`289` - Generator Expressions

      Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts

      steered by Hye-Shik Chang.

.. ======================================================================

PEP 292: Simpler String Substitutions

=====================================

Some new classes in the standard library provide an alternative mechanism for

substituting variables into strings; this style of substitution may be better

for applications where untrained users need to edit templates.

The usual way of substituting variables by name is the ``%`` operator::

   >>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}

   '2: The Best of Times'

When writing the template string, it can be easy to forget the ``i`` or ``s``

after the closing parenthesis.  This isn't a big problem if the template is in a

Python module, because you run the code, get an "Unsupported format character"

:exc:`ValueError`, and fix the problem.  However, consider an application such

as Mailman where template strings or translations are being edited by users who

aren't aware of the Python language.  The format string's syntax is complicated

to explain to such users, and if they make a mistake, it's difficult to provide

helpful feedback to them.

PEP 292 adds a :class:`Template` class to the :mod:`string` module that uses

``$`` to indicate a substitution::

   >>> import string

   >>> t = string.Template('$page: $title')

   >>> t.substitute({'page':2, 'title': 'The Best of Times'})

   '2: The Best of Times'

If a key is missing from the dictionary, the :meth:`substitute` method will

raise a :exc:`KeyError`.  There's also a :meth:`safe_substitute` method that

ignores missing keys::

   >>> t = string.Template('$page: $title')

   >>> t.safe_substitute({'page':3})

   '3: $title'

.. seealso::

   :pep:`292` - Simpler String Substitutions

      Written and implemented  by Barry Warsaw.

.. ======================================================================

PEP 318: Decorators for Functions and Methods

=============================================

Python 2.2 extended Python's object model by adding static methods and class

methods, but it didn't extend Python's syntax to provide any new way of defining

static or class methods.  Instead, you had to write a :keyword:`def` statement

in the usual way, and pass the resulting method to a :func:`staticmethod` or

:func:`classmethod` function that would wrap up the function as a method of the

new type. Your code would look like this::

   class C:

      def meth (cls):

          ...

      meth = classmethod(meth)   # Rebind name to wrapped-up class method

If the method was very long, it would be easy to miss or forget the

:func:`classmethod` invocation after the function body.

The intention was always to add some syntax to make such definitions more

readable, but at the time of 2.2's release a good syntax was not obvious.  Today

a good syntax *still* isn't obvious but users are asking for easier access to

the feature; a new syntactic feature has been added to meet this need.

The new feature is called "function decorators".  The name comes from the idea

that :func:`classmethod`, :func:`staticmethod`, and friends are storing

additional information on a function object; they're *decorating* functions with

more details.

The notation borrows from Java and uses the ``'@'`` character as an indicator.

Using the new syntax, the example above would be written::

   class C:

      @classmethod

      def meth (cls):

          ...

The ``@classmethod`` is shorthand for the ``meth=classmethod(meth)`` assignment.

More generally, if you have the following::

   @A

   @B

   @C

   def f ():

       ...

It's equivalent to the following pre-decorator code::

   def f(): ...

   f = A(B(C(f)))

Decorators must come on the line before a function definition, one decorator per

line, and can't be on the same line as the def statement, meaning that ``@A def

f(): ...`` is illegal.  You can only decorate function definitions, either at

the module level or inside a class; you can't decorate class definitions.

A decorator is just a function that takes the function to be decorated as an

argument and returns either the same function or some new object.  The return

value of the decorator need not be callable (though it typically is), unless

further decorators will be applied to the result.  It's easy to write your own

decorators.  The following simple example just sets an attribute on the function

object::

   >>> def deco(func):

   ...    func.attr = 'decorated'

   ...    return func

   ...

   >>> @deco

   ... def f(): pass

   ...

   >>> f

   <function f at 0x402ef0d4>

   >>> f.attr

   'decorated'

   >>>

As a slightly more realistic example, the following decorator checks that the

supplied argument is an integer::

   def require_int (func):

       def wrapper (arg):

           assert isinstance(arg, int)

           return func(arg)

       return wrapper

   @require_int

   def p1 (arg):

       print arg

   @require_int

   def p2(arg):

       print arg*2

An example in :pep:`318` contains a fancier version of this idea that lets you

both specify the required type and check the returned type.

Decorator functions can take arguments.  If arguments are supplied, your

decorator function is called with only those arguments and must return a new

decorator function; this function must take a single function and return a

function, as previously described.  In other words, ``@A @B @C(args)`` becomes::

   def f(): ...

   _deco = C(args)

   f = A(B(_deco(f)))

Getting this right can be slightly brain-bending, but it's not too difficult.

A small related change makes the :attr:`func_name` attribute of functions

writable.  This attribute is used to display function names in tracebacks, so

decorators should change the name of any new function that's constructed and

returned.

.. seealso::

   :pep:`318` - Decorators for Functions, Methods and Classes

      Written  by Kevin D. Smith, Jim Jewett, and Skip Montanaro.  Several people

      wrote patches implementing function decorators, but the one that was actually

      checked in was patch #979728, written by Mark Russell.

   http://www.python.org/moin/PythonDecoratorLibrary

      This Wiki page contains several examples of decorators.

.. ======================================================================

PEP 322: Reverse Iteration

==========================

A new built-in function, :func:`reversed(seq)`, takes a sequence and returns an

iterator that loops over the elements of the sequence  in reverse order.   ::

   >>> for i in reversed(xrange(1,4)):

   ...    print i

   ... 

   3

   2

   1

Compared to extended slicing, such as ``range(1,4)[::-1]``, :func:`reversed` is

easier to read, runs faster, and uses substantially less memory.

Note that :func:`reversed` only accepts sequences, not arbitrary iterators.  If

you want to reverse an iterator, first convert it to  a list with :func:`list`.

::

   >>> input = open('/etc/passwd', 'r')

   >>> for line in reversed(list(input)):

   ...   print line

   ... 

   root:*:0:0:System Administrator:/var/root:/bin/tcsh

     ...

.. seealso::

   :pep:`322` - Reverse Iteration

      Written and implemented by Raymond Hettinger.

.. ======================================================================

PEP 324: New subprocess Module

==============================

The standard library provides a number of ways to execute a subprocess, offering

different features and different levels of complexity.

:func:`os.system(command)` is easy to use, but slow (it runs a shell process

which executes the command) and dangerous (you have to be careful about escaping

the shell's metacharacters).  The :mod:`popen2` module offers classes that can

capture standard output and standard error from the subprocess, but the naming

is confusing.  The :mod:`subprocess` module cleans  this up, providing a unified

interface that offers all the features you might need.

Instead of :mod:`popen2`'s collection of classes, :mod:`subprocess` contains a

single class called :class:`Popen`  whose constructor supports a number of

different keyword arguments. ::

   class Popen(args, bufsize=0, executable=None,

   	    stdin=None, stdout=None, stderr=None,

   	    preexec_fn=None, close_fds=False, shell=False,

   	    cwd=None, env=None, universal_newlines=False,

   	    startupinfo=None, creationflags=0):

*args* is commonly a sequence of strings that will be the arguments to the

program executed as the subprocess.  (If the *shell* argument is true, *args*

can be a string which will then be passed on to the shell for interpretation,

just as :func:`os.system` does.)

*stdin*, *stdout*, and *stderr* specify what the subprocess's input, output, and

error streams will be.  You can provide a file object or a file descriptor, or

you can use the constant ``subprocess.PIPE`` to create a pipe between the

subprocess and the parent.

The constructor has a number of handy options:

* *close_fds* requests that all file descriptors be closed before running the

  subprocess.

* *cwd* specifies the working directory in which the subprocess will be executed

  (defaulting to whatever the parent's working directory is).

* *env* is a dictionary specifying environment variables.

* *preexec_fn* is a function that gets called before the child is started.

* *universal_newlines* opens the child's input and output using Python's

  universal newline feature.

Once you've created the :class:`Popen` instance,  you can call its :meth:`wait`

method to pause until the subprocess has exited, :meth:`poll` to check if it's

exited without pausing,  or :meth:`communicate(data)` to send the string *data*

to the subprocess's standard input.   :meth:`communicate(data)`  then reads any

data that the subprocess has sent to its standard output  or standard error,

returning a tuple ``(stdout_data, stderr_data)``.

:func:`call` is a shortcut that passes its arguments along to the :class:`Popen`

constructor, waits for the command to complete, and returns the status code of

the subprocess.  It can serve as a safer analog to :func:`os.system`::

   sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])

   if sts == 0:

       # Success

       ...

   else:

       # dpkg returned an error

       ...

The command is invoked without use of the shell.  If you really do want to  use

the shell, you can add ``shell=True`` as a keyword argument and provide a string

instead of a sequence::

   sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)

The PEP takes various examples of shell and Python code and shows how they'd be

translated into Python code that uses :mod:`subprocess`.  Reading this section

of the PEP is highly recommended.

.. seealso::

   :pep:`324` - subprocess - New process module

      Written and implemented by Peter Åstrand, with assistance from Fredrik Lundh and

      others.

.. ======================================================================

PEP 327: Decimal Data Type

==========================

Python has always supported floating-point (FP) numbers, based on the underlying

C :ctype:`double` type, as a data type.  However, while most programming

languages provide a floating-point type, many people (even programmers) are

unaware that floating-point numbers don't represent certain decimal fractions

accurately.  The new :class:`Decimal` type can represent these fractions

accurately, up to a user-specified precision limit.

Why is Decimal needed?

----------------------

The limitations arise from the representation used for floating-point numbers.

FP numbers are made up of three components:

* The sign, which is positive or negative.

* The mantissa, which is a single-digit binary number   followed by a fractional

  part.  For example, ``1.01`` in base-2 notation is ``1 + 0/2 + 1/4``, or 1.25 in

  decimal notation.

* The exponent, which tells where the decimal point is located in the number

  represented.

For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in

binary), and an exponent of 0 (the decimal point doesn't need to be shifted).

The number 5 has the same sign and mantissa, but the exponent is 2 because the

mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 \* 4 equals

5.

Modern systems usually provide floating-point support that conforms to a

standard called IEEE 754.  C's :ctype:`double` type is usually implemented as a

64-bit IEEE 754 number, which uses 52 bits of space for the mantissa.  This

means that numbers can only be specified to 52 bits of precision.  If you're

trying to represent numbers whose expansion repeats endlessly, the expansion is

cut off after 52 bits. Unfortunately, most software needs to produce output in

base 10, and common fractions in base 10 are often repeating decimals in binary.

For example, 1.1 decimal is binary ``1.0001100110011 ...``; .1 = 1/16 + 1/32 +

1/256 plus an infinite number of additional terms.  IEEE 754 has to chop off

that infinitely repeated decimal after 52 digits, so the representation is

slightly inaccurate.

Sometimes you can see this inaccuracy when the number is printed::

   >>> 1.1

   1.1000000000000001

The inaccuracy isn't always visible when you print the number because the FP-to-

decimal-string conversion is provided by the C library, and most C libraries try

to produce sensible output.  Even if it's not displayed, however, the inaccuracy

is still there and subsequent operations can magnify the error.

For many applications this doesn't matter.  If I'm plotting points and

displaying them on my monitor, the difference between 1.1 and 1.1000000000000001

is too small to be visible.  Reports often limit output to a certain number of

decimal places, and if you round the number to two or three or even eight

decimal places, the error is never apparent.  However, for applications where it

does matter,  it's a lot of work to implement your own custom arithmetic

routines.