1 ****************************

     2   What's New in Python 2.4  

     3 ****************************

     4 

     5 :Author: A.M. Kuchling

     6 

     7 .. |release| replace:: 1.02

     8 

     9 .. $Id: whatsnew24.tex 54632 2007-03-31 11:59:54Z georg.brandl $

    10 .. Don't write extensive text for new sections; I'll do that.

    11 .. Feel free to add commented-out reminders of things that need

    12 .. to be covered.  --amk

    13 

    14 This article explains the new features in Python 2.4.1, released on March 30,

    15 2005.

    16 

    17 Python 2.4 is a medium-sized release.  It doesn't introduce as many changes as

    18 the radical Python 2.2, but introduces more features than the conservative 2.3

    19 release.  The most significant new language features are function decorators and

    20 generator expressions; most other changes are to the standard library.

    21 

    22 According to the CVS change logs, there were 481 patches applied and 502 bugs

    23 fixed between Python 2.3 and 2.4.  Both figures are likely to be underestimates.

    24 

    25 This article doesn't attempt to provide a complete specification of every single

    26 new feature, but instead provides a brief introduction to each feature.  For

    27 full details, you should refer to the documentation for Python 2.4, such as the

    28 Python Library Reference and the Python Reference Manual.  Often you will be

    29 referred to the PEP for a particular new feature for explanations of the

    30 implementation and design rationale.

    31 

    32 .. ======================================================================

    33 

    34 

    35 PEP 218: Built-In Set Objects

    36 =============================

    37 

    38 Python 2.3 introduced the :mod:`sets` module.  C implementations of set data

    39 types have now been added to the Python core as two new built-in types,

    40 :func:`set(iterable)` and :func:`frozenset(iterable)`.  They provide high speed

    41 operations for membership testing, for eliminating duplicates from sequences,

    42 and for mathematical operations like unions, intersections, differences, and

    43 symmetric differences. ::

    44 

    45    >>> a = set('abracadabra')              # form a set from a string

    46    >>> 'z' in a                            # fast membership testing

    47    False

    48    >>> a                                   # unique letters in a

    49    set(['a', 'r', 'b', 'c', 'd'])

    50    >>> ''.join(a)                          # convert back into a string

    51    'arbcd'

    52 

    53    >>> b = set('alacazam')                 # form a second set

    54    >>> a - b                               # letters in a but not in b

    55    set(['r', 'd', 'b'])

    56    >>> a | b                               # letters in either a or b

    57    set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])

    58    >>> a & b                               # letters in both a and b

    59    set(['a', 'c'])

    60    >>> a ^ b                               # letters in a or b but not both

    61    set(['r', 'd', 'b', 'm', 'z', 'l'])

    62 

    63    >>> a.add('z')                          # add a new element

    64    >>> a.update('wxy')                     # add multiple new elements

    65    >>> a

    66    set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])       

    67    >>> a.remove('x')                       # take one element out

    68    >>> a

    69    set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])       

    70 

    71 The :func:`frozenset` type is an immutable version of :func:`set`. Since it is

    72 immutable and hashable, it may be used as a dictionary key or as a member of

    73 another set.

    74 

    75 The :mod:`sets` module remains in the standard library, and may be useful if you

    76 wish to subclass the :class:`Set` or :class:`ImmutableSet` classes.  There are

    77 currently no plans to deprecate the module.

    78 

    79 

    80 .. seealso::

    81 

    82    :pep:`218` - Adding a Built-In Set Object Type

    83       Originally proposed by Greg Wilson and ultimately implemented by Raymond

    84       Hettinger.

    85 

    86 .. ======================================================================

    87 

    88 

    89 PEP 237: Unifying Long Integers and Integers

    90 ============================================

    91 

    92 The lengthy transition process for this PEP, begun in Python 2.2, takes another

    93 step forward in Python 2.4.  In 2.3, certain integer operations that would

    94 behave differently after int/long unification triggered :exc:`FutureWarning`

    95 warnings and returned values limited to 32 or 64 bits (depending on your

    96 platform).  In 2.4, these expressions no longer produce a warning and instead

    97 produce a different result that's usually a long integer.

    98 

    99 The problematic expressions are primarily left shifts and lengthy hexadecimal

   100 and octal constants.  For example, ``2 << 32`` results in a warning in 2.3,

   101 evaluating to 0 on 32-bit platforms.  In Python 2.4, this expression now returns

   102 the correct answer, 8589934592.

   103 

   104 

   105 .. seealso::

   106 

   107    :pep:`237` - Unifying Long Integers and Integers

   108       Original PEP written by Moshe Zadka and GvR.  The changes for 2.4 were

   109       implemented by  Kalle Svensson.

   110 

   111 .. ======================================================================

   112 

   113 

   114 PEP 289: Generator Expressions

   115 ==============================

   116 

   117 The iterator feature introduced in Python 2.2 and the :mod:`itertools` module

   118 make it easier to write programs that loop through large data sets without

   119 having the entire data set in memory at one time.  List comprehensions don't fit

   120 into this picture very well because they produce a Python list object containing

   121 all of the items.  This unavoidably pulls all of the objects into memory, which

   122 can be a problem if your data set is very large.  When trying to write a

   123 functionally-styled program, it would be natural to write something like::

   124 

   125    links = [link for link in get_all_links() if not link.followed]

   126    for link in links:

   127        ...

   128 

   129 instead of  ::

   130 

   131    for link in get_all_links():

   132        if link.followed:

   133            continue

   134        ...

   135 

   136 The first form is more concise and perhaps more readable, but if you're dealing

   137 with a large number of link objects you'd have to write the second form to avoid

   138 having all link objects in memory at the same time.

   139 

   140 Generator expressions work similarly to list comprehensions but don't

   141 materialize the entire list; instead they create a generator that will return

   142 elements one by one.  The above example could be written as::

   143 

   144    links = (link for link in get_all_links() if not link.followed)

   145    for link in links:

   146        ...

   147 

   148 Generator expressions always have to be written inside parentheses, as in the

   149 above example.  The parentheses signalling a function call also count, so if you

   150 want to create an iterator that will be immediately passed to a function you

   151 could write::

   152 

   153    print sum(obj.count for obj in list_all_objects())

   154 

   155 Generator expressions differ from list comprehensions in various small ways.

   156 Most notably, the loop variable (*obj* in the above example) is not accessible

   157 outside of the generator expression.  List comprehensions leave the variable

   158 assigned to its last value; future versions of Python will change this, making

   159 list comprehensions match generator expressions in this respect.

   160 

   161 

   162 .. seealso::

   163 

   164    :pep:`289` - Generator Expressions

   165       Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts

   166       steered by Hye-Shik Chang.

   167 

   168 .. ======================================================================

   169 

   170 

   171 PEP 292: Simpler String Substitutions

   172 =====================================

   173 

   174 Some new classes in the standard library provide an alternative mechanism for

   175 substituting variables into strings; this style of substitution may be better

   176 for applications where untrained users need to edit templates.

   177 

   178 The usual way of substituting variables by name is the ``%`` operator::

   179 

   180    >>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}

   181    '2: The Best of Times'

   182 

   183 When writing the template string, it can be easy to forget the ``i`` or ``s``

   184 after the closing parenthesis.  This isn't a big problem if the template is in a

   185 Python module, because you run the code, get an "Unsupported format character"

   186 :exc:`ValueError`, and fix the problem.  However, consider an application such

   187 as Mailman where template strings or translations are being edited by users who

   188 aren't aware of the Python language.  The format string's syntax is complicated

   189 to explain to such users, and if they make a mistake, it's difficult to provide

   190 helpful feedback to them.

   191 

   192 PEP 292 adds a :class:`Template` class to the :mod:`string` module that uses

   193 ``$`` to indicate a substitution::

   194 

   195    >>> import string

   196    >>> t = string.Template('$page: $title')

   197    >>> t.substitute({'page':2, 'title': 'The Best of Times'})

   198    '2: The Best of Times'

   199 

   200 If a key is missing from the dictionary, the :meth:`substitute` method will

   201 raise a :exc:`KeyError`.  There's also a :meth:`safe_substitute` method that

   202 ignores missing keys::

   203 

   204    >>> t = string.Template('$page: $title')

   205    >>> t.safe_substitute({'page':3})

   206    '3: $title'

   207 

   208 

   209 .. seealso::

   210 

   211    :pep:`292` - Simpler String Substitutions

   212       Written and implemented  by Barry Warsaw.

   213 

   214 .. ======================================================================

   215 

   216 

   217 PEP 318: Decorators for Functions and Methods

   218 =============================================

   219 

   220 Python 2.2 extended Python's object model by adding static methods and class

   221 methods, but it didn't extend Python's syntax to provide any new way of defining

   222 static or class methods.  Instead, you had to write a :keyword:`def` statement

   223 in the usual way, and pass the resulting method to a :func:`staticmethod` or

   224 :func:`classmethod` function that would wrap up the function as a method of the

   225 new type. Your code would look like this::

   226 

   227    class C:

   228       def meth (cls):

   229           ...

   230 

   231       meth = classmethod(meth)   # Rebind name to wrapped-up class method

   232 

   233 If the method was very long, it would be easy to miss or forget the

   234 :func:`classmethod` invocation after the function body.

   235 

   236 The intention was always to add some syntax to make such definitions more

   237 readable, but at the time of 2.2's release a good syntax was not obvious.  Today

   238 a good syntax *still* isn't obvious but users are asking for easier access to

   239 the feature; a new syntactic feature has been added to meet this need.

   240 

   241 The new feature is called "function decorators".  The name comes from the idea

   242 that :func:`classmethod`, :func:`staticmethod`, and friends are storing

   243 additional information on a function object; they're *decorating* functions with

   244 more details.

   245 

   246 The notation borrows from Java and uses the ``'@'`` character as an indicator.

   247 Using the new syntax, the example above would be written::

   248 

   249    class C:

   250 

   251       @classmethod

   252       def meth (cls):

   253           ...

   254 

   255 

   256 The ``@classmethod`` is shorthand for the ``meth=classmethod(meth)`` assignment.

   257 More generally, if you have the following::

   258 

   259    @A

   260    @B

   261    @C

   262    def f ():

   263        ...

   264 

   265 It's equivalent to the following pre-decorator code::

   266 

   267    def f(): ...

   268    f = A(B(C(f)))

   269 

   270 Decorators must come on the line before a function definition, one decorator per

   271 line, and can't be on the same line as the def statement, meaning that ``@A def

   272 f(): ...`` is illegal.  You can only decorate function definitions, either at

   273 the module level or inside a class; you can't decorate class definitions.

   274 

   275 A decorator is just a function that takes the function to be decorated as an

   276 argument and returns either the same function or some new object.  The return

   277 value of the decorator need not be callable (though it typically is), unless

   278 further decorators will be applied to the result.  It's easy to write your own

   279 decorators.  The following simple example just sets an attribute on the function

   280 object::

   281 

   282    >>> def deco(func):

   283    ...    func.attr = 'decorated'

   284    ...    return func

   285    ...

   286    >>> @deco

   287    ... def f(): pass

   288    ...

   289    >>> f

   290    <function f at 0x402ef0d4>

   291    >>> f.attr

   292    'decorated'

   293    >>>

   294 

   295 As a slightly more realistic example, the following decorator checks that the

   296 supplied argument is an integer::

   297 

   298    def require_int (func):

   299        def wrapper (arg):

   300            assert isinstance(arg, int)

   301            return func(arg)

   302 

   303        return wrapper

   304 

   305    @require_int

   306    def p1 (arg):

   307        print arg

   308 

   309    @require_int

   310    def p2(arg):

   311        print arg*2

   312 

   313 An example in :pep:`318` contains a fancier version of this idea that lets you

   314 both specify the required type and check the returned type.

   315 

   316 Decorator functions can take arguments.  If arguments are supplied, your

   317 decorator function is called with only those arguments and must return a new

   318 decorator function; this function must take a single function and return a

   319 function, as previously described.  In other words, ``@A @B @C(args)`` becomes::

   320 

   321    def f(): ...

   322    _deco = C(args)

   323    f = A(B(_deco(f)))

   324 

   325 Getting this right can be slightly brain-bending, but it's not too difficult.

   326 

   327 A small related change makes the :attr:`func_name` attribute of functions

   328 writable.  This attribute is used to display function names in tracebacks, so

   329 decorators should change the name of any new function that's constructed and

   330 returned.

   331 

   332 

   333 .. seealso::

   334 

   335    :pep:`318` - Decorators for Functions, Methods and Classes

   336       Written  by Kevin D. Smith, Jim Jewett, and Skip Montanaro.  Several people

   337       wrote patches implementing function decorators, but the one that was actually

   338       checked in was patch #979728, written by Mark Russell.

   339 

   340    http://www.python.org/moin/PythonDecoratorLibrary

   341       This Wiki page contains several examples of decorators.

   342 

   343 .. ======================================================================

   344 

   345 

   346 PEP 322: Reverse Iteration

   347 ==========================

   348 

   349 A new built-in function, :func:`reversed(seq)`, takes a sequence and returns an

   350 iterator that loops over the elements of the sequence  in reverse order.   ::

   351 

   352    >>> for i in reversed(xrange(1,4)):

   353    ...    print i

   354    ... 

   355    3

   356    2

   357    1

   358 

   359 Compared to extended slicing, such as ``range(1,4)[::-1]``, :func:`reversed` is

   360 easier to read, runs faster, and uses substantially less memory.

   361 

   362 Note that :func:`reversed` only accepts sequences, not arbitrary iterators.  If

   363 you want to reverse an iterator, first convert it to  a list with :func:`list`.

   364 ::

   365 

   366    >>> input = open('/etc/passwd', 'r')

   367    >>> for line in reversed(list(input)):

   368    ...   print line

   369    ... 

   370    root:*:0:0:System Administrator:/var/root:/bin/tcsh

   371      ...

   372 

   373 

   374 .. seealso::

   375 

   376    :pep:`322` - Reverse Iteration

   377       Written and implemented by Raymond Hettinger.

   378 

   379 .. ======================================================================

   380 

   381 

   382 PEP 324: New subprocess Module

   383 ==============================

   384 

   385 The standard library provides a number of ways to execute a subprocess, offering

   386 different features and different levels of complexity.

   387 :func:`os.system(command)` is easy to use, but slow (it runs a shell process

   388 which executes the command) and dangerous (you have to be careful about escaping

   389 the shell's metacharacters).  The :mod:`popen2` module offers classes that can

   390 capture standard output and standard error from the subprocess, but the naming

   391 is confusing.  The :mod:`subprocess` module cleans  this up, providing a unified

   392 interface that offers all the features you might need.

   393 

   394 Instead of :mod:`popen2`'s collection of classes, :mod:`subprocess` contains a

   395 single class called :class:`Popen`  whose constructor supports a number of

   396 different keyword arguments. ::

   397 

   398    class Popen(args, bufsize=0, executable=None,

   399    	    stdin=None, stdout=None, stderr=None,

   400    	    preexec_fn=None, close_fds=False, shell=False,

   401    	    cwd=None, env=None, universal_newlines=False,

   402    	    startupinfo=None, creationflags=0):

   403 

   404 *args* is commonly a sequence of strings that will be the arguments to the

   405 program executed as the subprocess.  (If the *shell* argument is true, *args*

   406 can be a string which will then be passed on to the shell for interpretation,

   407 just as :func:`os.system` does.)

   408 

   409 *stdin*, *stdout*, and *stderr* specify what the subprocess's input, output, and

   410 error streams will be.  You can provide a file object or a file descriptor, or

   411 you can use the constant ``subprocess.PIPE`` to create a pipe between the

   412 subprocess and the parent.

   413 

   414 The constructor has a number of handy options:

   415 

   416 * *close_fds* requests that all file descriptors be closed before running the

   417   subprocess.

   418 

   419 * *cwd* specifies the working directory in which the subprocess will be executed

   420   (defaulting to whatever the parent's working directory is).

   421 

   422 * *env* is a dictionary specifying environment variables.

   423 

   424 * *preexec_fn* is a function that gets called before the child is started.

   425 

   426 * *universal_newlines* opens the child's input and output using Python's

   427   universal newline feature.

   428 

   429 Once you've created the :class:`Popen` instance,  you can call its :meth:`wait`

   430 method to pause until the subprocess has exited, :meth:`poll` to check if it's

   431 exited without pausing,  or :meth:`communicate(data)` to send the string *data*

   432 to the subprocess's standard input.   :meth:`communicate(data)`  then reads any

   433 data that the subprocess has sent to its standard output  or standard error,

   434 returning a tuple ``(stdout_data, stderr_data)``.

   435 

   436 :func:`call` is a shortcut that passes its arguments along to the :class:`Popen`

   437 constructor, waits for the command to complete, and returns the status code of

   438 the subprocess.  It can serve as a safer analog to :func:`os.system`::

   439 

   440    sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])

   441    if sts == 0:

   442        # Success

   443        ...

   444    else:

   445        # dpkg returned an error

   446        ...

   447 

   448 The command is invoked without use of the shell.  If you really do want to  use

   449 the shell, you can add ``shell=True`` as a keyword argument and provide a string

   450 instead of a sequence::

   451 

   452    sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)

   453 

   454 The PEP takes various examples of shell and Python code and shows how they'd be

   455 translated into Python code that uses :mod:`subprocess`.  Reading this section

   456 of the PEP is highly recommended.

   457 

   458 

   459 .. seealso::

   460 

   461    :pep:`324` - subprocess - New process module

   462       Written and implemented by Peter Åstrand, with assistance from Fredrik Lundh and

   463       others.

   464 

   465 .. ======================================================================

   466 

   467 

   468 PEP 327: Decimal Data Type

   469 ==========================

   470 

   471 Python has always supported floating-point (FP) numbers, based on the underlying

   472 C :ctype:`double` type, as a data type.  However, while most programming

   473 languages provide a floating-point type, many people (even programmers) are

   474 unaware that floating-point numbers don't represent certain decimal fractions

   475 accurately.  The new :class:`Decimal` type can represent these fractions

   476 accurately, up to a user-specified precision limit.

   477 

   478 

   479 Why is Decimal needed?

   480 ----------------------

   481 

   482 The limitations arise from the representation used for floating-point numbers.

   483 FP numbers are made up of three components:

   484 

   485 * The sign, which is positive or negative.

   486 

   487 * The mantissa, which is a single-digit binary number   followed by a fractional

   488   part.  For example, ``1.01`` in base-2 notation is ``1 + 0/2 + 1/4``, or 1.25 in

   489   decimal notation.

   490 

   491 * The exponent, which tells where the decimal point is located in the number

   492   represented.

   493 

   494 For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in

   495 binary), and an exponent of 0 (the decimal point doesn't need to be shifted).

   496 The number 5 has the same sign and mantissa, but the exponent is 2 because the

   497 mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 \* 4 equals

   498 5.

   499 

   500 Modern systems usually provide floating-point support that conforms to a

   501 standard called IEEE 754.  C's :ctype:`double` type is usually implemented as a

   502 64-bit IEEE 754 number, which uses 52 bits of space for the mantissa.  This

   503 means that numbers can only be specified to 52 bits of precision.  If you're

   504 trying to represent numbers whose expansion repeats endlessly, the expansion is

   505 cut off after 52 bits. Unfortunately, most software needs to produce output in

   506 base 10, and common fractions in base 10 are often repeating decimals in binary.

   507 For example, 1.1 decimal is binary ``1.0001100110011 ...``; .1 = 1/16 + 1/32 +

   508 1/256 plus an infinite number of additional terms.  IEEE 754 has to chop off

   509 that infinitely repeated decimal after 52 digits, so the representation is

   510 slightly inaccurate.

   511 

   512 Sometimes you can see this inaccuracy when the number is printed::

   513 

   514    >>> 1.1

   515    1.1000000000000001

   516 

   517 The inaccuracy isn't always visible when you print the number because the FP-to-

   518 decimal-string conversion is provided by the C library, and most C libraries try

   519 to produce sensible output.  Even if it's not displayed, however, the inaccuracy

   520 is still there and subsequent operations can magnify the error.

   521 

   522 For many applications this doesn't matter.  If I'm plotting points and

   523 displaying them on my monitor, the difference between 1.1 and 1.1000000000000001

   524 is too small to be visible.  Reports often limit output to a certain number of

   525 decimal places, and if you round the number to two or three or even eight

   526 decimal places, the error is never apparent.  However, for applications where it

   527 does matter,  it's a lot of work to implement your own custom arithmetic

   528 routines.

   529 

   530 Hence, the :class:`Decimal` type was created.

   531 

   532 

   533 The :class:`Decimal` type

   534 -------------------------

   535 

   536 A new module, :mod:`decimal`, was added to Python's standard library.  It

   537 contains two classes, :class:`Decimal` and :class:`Context`.  :class:`Decimal`

   538 instances represent numbers, and :class:`Context` instances are used to wrap up

   539 various settings such as the precision and default rounding mode.

   540 

   541 :class:`Decimal` instances are immutable, like regular Python integers and FP

   542 numbers; once it's been created, you can't change the value an instance

   543 represents.  :class:`Decimal` instances can be created from integers or

   544 strings::

   545 

   546    >>> import decimal

   547    >>> decimal.Decimal(1972)

   548    Decimal("1972")

   549    >>> decimal.Decimal("1.1")

   550    Decimal("1.1")

   551 

   552 You can also provide tuples containing the sign, the mantissa represented  as a

   553 tuple of decimal digits, and the exponent::

   554 

   555    >>> decimal.Decimal((1, (1, 4, 7, 5), -2))

   556    Decimal("-14.75")

   557 

   558 Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is

   559 negative.

   560 

   561 Converting from floating-point numbers poses a bit of a problem: should the FP

   562 number representing 1.1 turn into the decimal number for exactly 1.1, or for 1.1

   563 plus whatever inaccuracies are introduced? The decision was to dodge the issue

   564 and leave such a conversion out of the API.  Instead, you should convert the

   565 floating-point number into a string using the desired precision and pass the

   566 string to the :class:`Decimal` constructor::

   567 

   568    >>> f = 1.1

   569    >>> decimal.Decimal(str(f))

   570    Decimal("1.1")

   571    >>> decimal.Decimal('%.12f' % f)

   572    Decimal("1.100000000000")

   573 

   574 Once you have :class:`Decimal` instances, you can perform the usual mathematical

   575 operations on them.  One limitation: exponentiation requires an integer

   576 exponent::

   577 

   578    >>> a = decimal.Decimal('35.72')

   579    >>> b = decimal.Decimal('1.73')

   580    >>> a+b

   581    Decimal("37.45")

   582    >>> a-b

   583    Decimal("33.99")

   584    >>> a*b

   585    Decimal("61.7956")

   586    >>> a/b

   587    Decimal("20.64739884393063583815028902")

   588    >>> a ** 2

   589    Decimal("1275.9184")

   590    >>> a**b

   591    Traceback (most recent call last):

   592      ...

   593    decimal.InvalidOperation: x ** (non-integer)

   594 

   595 You can combine :class:`Decimal` instances with integers, but not with floating-

   596 point numbers::

   597 

   598    >>> a + 4

   599    Decimal("39.72")

   600    >>> a + 4.5

   601    Traceback (most recent call last):

   602      ...

   603    TypeError: You can interact Decimal only with int, long or Decimal data types.

   604    >>>

   605 

   606 :class:`Decimal` numbers can be used with the :mod:`math` and :mod:`cmath`

   607 modules, but note that they'll be immediately converted to  floating-point

   608 numbers before the operation is performed, resulting in a possible loss of

   609 precision and accuracy.  You'll also get back a regular floating-point number

   610 and not a :class:`Decimal`.   ::

   611 

   612    >>> import math, cmath

   613    >>> d = decimal.Decimal('123456789012.345')

   614    >>> math.sqrt(d)

   615    351364.18288201344

   616    >>> cmath.sqrt(-d)

   617    351364.18288201344j

   618 

   619 :class:`Decimal` instances have a :meth:`sqrt` method that returns a

   620 :class:`Decimal`, but if you need other things such as trigonometric functions

   621 you'll have to implement them. ::

   622 

   623    >>> d.sqrt()

   624    Decimal("351364.1828820134592177245001")

   625 

   626 

   627 The :class:`Context` type

   628 -------------------------

   629 

   630 Instances of the :class:`Context` class encapsulate several settings for

   631 decimal operations:

   632 

   633 * :attr:`prec` is the precision, the number of decimal places.

   634 

   635 * :attr:`rounding` specifies the rounding mode.  The :mod:`decimal` module has

   636   constants for the various possibilities: :const:`ROUND_DOWN`,

   637   :const:`ROUND_CEILING`,  :const:`ROUND_HALF_EVEN`, and various others.

   638 

   639 * :attr:`traps` is a dictionary specifying what happens on encountering certain

   640   error conditions: either  an exception is raised or  a value is returned.  Some

   641   examples of error conditions are division by zero, loss of precision, and

   642   overflow.

   643 

   644 There's a thread-local default context available by calling :func:`getcontext`;

   645 you can change the properties of this context to alter the default precision,

   646 rounding, or trap handling.  The following example shows the effect of changing

   647 the precision of the default context::

   648 

   649    >>> decimal.getcontext().prec

   650    28

   651    >>> decimal.Decimal(1) / decimal.Decimal(7)

   652    Decimal("0.1428571428571428571428571429")

   653    >>> decimal.getcontext().prec = 9 

   654    >>> decimal.Decimal(1) / decimal.Decimal(7)

   655    Decimal("0.142857143")

   656 

   657 The default action for error conditions is selectable; the module can either

   658 return a special value such as infinity or not-a-number, or exceptions can be

   659 raised::

   660 

   661    >>> decimal.Decimal(1) / decimal.Decimal(0)

   662    Traceback (most recent call last):

   663      ...

   664    decimal.DivisionByZero: x / 0

   665    >>> decimal.getcontext().traps[decimal.DivisionByZero] = False

   666    >>> decimal.Decimal(1) / decimal.Decimal(0)

   667    Decimal("Infinity")

   668    >>> 

   669 

   670 The :class:`Context` instance also has various methods for formatting  numbers

   671 such as :meth:`to_eng_string` and :meth:`to_sci_string`.

   672 

   673 For more information, see the documentation for the :mod:`decimal` module, which

   674 includes a quick-start tutorial and a reference.

   675 

   676 

   677 .. seealso::

   678 

   679    :pep:`327` - Decimal Data Type

   680       Written by Facundo Batista and implemented by Facundo Batista, Eric Price,

   681       Raymond Hettinger, Aahz, and Tim Peters.

   682 

   683    http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html

   684       A more detailed overview of the IEEE-754 representation.

   685 

   686    http://www.lahey.com/float.htm

   687       The article uses Fortran code to illustrate many of the problems that floating-

   688       point inaccuracy can cause.

   689 

   690    http://www2.hursley.ibm.com/decimal/

   691       A description of a decimal-based representation.  This representation is being

   692       proposed as a standard, and underlies the new Python decimal type.  Much of this

   693       material was written by Mike Cowlishaw, designer of the Rexx language.

   694 

   695 .. ======================================================================

   696 

   697 

   698 PEP 328: Multi-line Imports

   699 ===========================

   700 

   701 One language change is a small syntactic tweak aimed at making it easier to

   702 import many names from a module.  In a ``from module import names`` statement,

   703 *names* is a sequence of names separated by commas.  If the sequence is  very

   704 long, you can either write multiple imports from the same module, or you can use

   705 backslashes to escape the line endings like this::

   706 

   707    from SimpleXMLRPCServer import SimpleXMLRPCServer,\

   708                SimpleXMLRPCRequestHandler,\

   709                CGIXMLRPCRequestHandler,\

   710                resolve_dotted_attribute

   711 

   712 The syntactic change in Python 2.4 simply allows putting the names within

   713 parentheses.  Python ignores newlines within a parenthesized expression, so the

   714 backslashes are no longer needed::

   715 

   716    from SimpleXMLRPCServer import (SimpleXMLRPCServer,

   717                                    SimpleXMLRPCRequestHandler,

   718                                    CGIXMLRPCRequestHandler,

   719                                    resolve_dotted_attribute)

   720 

   721 The PEP also proposes that all :keyword:`import` statements be absolute imports,

   722 with a leading ``.`` character to indicate a relative import.  This part of the

   723 PEP was not implemented for Python 2.4, but was completed for Python 2.5.

   724 

   725 

   726 .. seealso::

   727 

   728    :pep:`328` - Imports: Multi-Line and Absolute/Relative

   729       Written by Aahz.  Multi-line imports were implemented by Dima Dorfman.

   730 

   731 .. ======================================================================

   732 

   733 

   734 PEP 331: Locale-Independent Float/String Conversions

   735 ====================================================

   736 

   737 The :mod:`locale` modules lets Python software select various conversions and

   738 display conventions that are localized to a particular country or language.

   739 However, the module was careful to not change the numeric locale because various

   740 functions in Python's implementation required that the numeric locale remain set

   741 to the ``'C'`` locale.  Often this was because the code was using the C

   742 library's :cfunc:`atof` function.

   743 

   744 Not setting the numeric locale caused trouble for extensions that used third-

   745 party C libraries, however, because they wouldn't have the correct locale set.

   746 The motivating example was GTK+, whose user interface widgets weren't displaying

   747 numbers in the current locale.

   748 

   749 The solution described in the PEP is to add three new functions to the Python

   750 API that perform ASCII-only conversions, ignoring the locale setting:

   751 

   752 * :cfunc:`PyOS_ascii_strtod(str, ptr)`  and :cfunc:`PyOS_ascii_atof(str, ptr)`

   753   both convert a string to a C :ctype:`double`.

   754 

   755 * :cfunc:`PyOS_ascii_formatd(buffer, buf_len, format, d)` converts a

   756   :ctype:`double` to an ASCII string.

   757 

   758 The code for these functions came from the GLib library

   759 (http://developer.gnome.org/arch/gtk/glib.html), whose developers kindly

   760 relicensed the relevant functions and donated them to the Python Software

   761 Foundation.  The :mod:`locale` module  can now change the numeric locale,

   762 letting extensions such as GTK+  produce the correct results.

   763 

   764 

   765 .. seealso::

   766 

   767    :pep:`331` - Locale-Independent Float/String Conversions

   768       Written by Christian R. Reis, and implemented by Gustavo Carneiro.

   769 

   770 .. ======================================================================

   771 

   772 

   773 Other Language Changes

   774 ======================

   775 

   776 Here are all of the changes that Python 2.4 makes to the core Python language.

   777 

   778 * Decorators for functions and methods were added (:pep:`318`).

   779 

   780 * Built-in :func:`set` and :func:`frozenset` types were  added (:pep:`218`).

   781   Other new built-ins include the :func:`reversed(seq)` function (:pep:`322`).

   782 

   783 * Generator expressions were added (:pep:`289`).

   784 

   785 * Certain numeric expressions no longer return values restricted to 32 or 64

   786   bits (:pep:`237`).

   787 

   788 * You can now put parentheses around the list of names in a ``from module import

   789   names`` statement (:pep:`328`).

   790 

   791 * The :meth:`dict.update` method now accepts the same argument forms as the

   792   :class:`dict` constructor.  This includes any mapping, any iterable of key/value

   793   pairs, and keyword arguments. (Contributed by Raymond Hettinger.)

   794 

   795 * The string methods :meth:`ljust`, :meth:`rjust`, and :meth:`center` now take

   796   an optional argument for specifying a fill character other than a space.

   797   (Contributed by Raymond Hettinger.)

   798 

   799 * Strings also gained an :meth:`rsplit` method that works like the :meth:`split`

   800   method but splits from the end of the string.   (Contributed by Sean

   801   Reifschneider.) ::

   802 

   803      >>> 'www.python.org'.split('.', 1)

   804      ['www', 'python.org']

   805      'www.python.org'.rsplit('.', 1)

   806      ['www.python', 'org']        

   807 

   808 * Three keyword parameters, *cmp*, *key*, and *reverse*, were added to the

   809   :meth:`sort` method of lists. These parameters make some common usages of

   810   :meth:`sort` simpler. All of these parameters are optional.

   811 

   812   For the *cmp* parameter, the value should be a comparison function that takes

   813   two parameters and returns -1, 0, or +1 depending on how the parameters compare.

   814   This function will then be used to sort the list.  Previously this was the only

   815   parameter that could be provided to :meth:`sort`.

   816 

   817   *key* should be a single-parameter function that takes a list element and

   818   returns a comparison key for the element.  The list is then sorted using the

   819   comparison keys.  The following example sorts a list case-insensitively::

   820 

   821      >>> L = ['A', 'b', 'c', 'D']

   822      >>> L.sort()                 # Case-sensitive sort

   823      >>> L

   824      ['A', 'D', 'b', 'c']

   825      >>> # Using 'key' parameter to sort list

   826      >>> L.sort(key=lambda x: x.lower())

   827      >>> L

   828      ['A', 'b', 'c', 'D']

   829      >>> # Old-fashioned way

   830      >>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))

   831      >>> L

   832      ['A', 'b', 'c', 'D']

   833 

   834   The last example, which uses the *cmp* parameter, is the old way to perform a

   835   case-insensitive sort.  It works but is slower than using a *key* parameter.

   836   Using *key* calls :meth:`lower` method once for each element in the list while

   837   using *cmp* will call it twice for each comparison, so using *key* saves on

   838   invocations of the :meth:`lower` method.

   839 

   840   For simple key functions and comparison functions, it is often possible to avoid

   841   a :keyword:`lambda` expression by using an unbound method instead.  For example,

   842   the above case-insensitive sort is best written as::

   843 

   844      >>> L.sort(key=str.lower)

   845      >>> L

   846      ['A', 'b', 'c', 'D']

   847 

   848   Finally, the *reverse* parameter takes a Boolean value.  If the value is true,

   849   the list will be sorted into reverse order. Instead of ``L.sort() ;

   850   L.reverse()``, you can now write ``L.sort(reverse=True)``.

   851 

   852   The results of sorting are now guaranteed to be stable.  This means that two

   853   entries with equal keys will be returned in the same order as they were input.

   854   For example, you can sort a list of people by name, and then sort the list by

   855   age, resulting in a list sorted by age where people with the same age are in

   856   name-sorted order.

   857 

   858   (All changes to :meth:`sort` contributed by Raymond Hettinger.)

   859 

   860 * There is a new built-in function :func:`sorted(iterable)` that works like the

   861   in-place :meth:`list.sort` method but can be used in expressions.  The

   862   differences are:

   863 

   864 * the input may be any iterable;

   865 

   866 * a newly formed copy is sorted, leaving the original intact; and

   867 

   868 * the expression returns the new sorted copy

   869 

   870   ::

   871 

   872      >>> L = [9,7,8,3,2,4,1,6,5]

   873      >>> [10+i for i in sorted(L)]       # usable in a list comprehension

   874      [11, 12, 13, 14, 15, 16, 17, 18, 19]

   875      >>> L                               # original is left unchanged

   876      [9,7,8,3,2,4,1,6,5]

   877      >>> sorted('Monty Python')          # any iterable may be an input

   878      [' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']

   879 

   880      >>> # List the contents of a dict sorted by key values

   881      >>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)

   882      >>> for k, v in sorted(colormap.iteritems()):

   883      ...     print k, v

   884      ...

   885      black 4

   886      blue 2

   887      green 3

   888      red 1

   889      yellow 5

   890 

   891   (Contributed by Raymond Hettinger.)

   892 

   893 * Integer operations will no longer trigger an :exc:`OverflowWarning`. The

   894   :exc:`OverflowWarning` warning will disappear in Python 2.5.

   895 

   896 * The interpreter gained a new switch, :option:`-m`, that takes a name, searches

   897   for the corresponding  module on ``sys.path``, and runs the module as a script.

   898   For example,  you can now run the Python profiler with ``python -m profile``.

   899   (Contributed by Nick Coghlan.)

   900 

   901 * The :func:`eval(expr, globals, locals)` and :func:`execfile(filename, globals,

   902   locals)` functions and the :keyword:`exec` statement now accept any mapping type

   903   for the *locals* parameter.  Previously this had to be a regular Python

   904   dictionary.  (Contributed by Raymond Hettinger.)

   905 

   906 * The :func:`zip` built-in function and :func:`itertools.izip` now return an

   907   empty list if called with no arguments. Previously they raised a

   908   :exc:`TypeError` exception.  This makes them more suitable for use with variable

   909   length argument lists::

   910 

   911      >>> def transpose(array):

   912      ...    return zip(*array)

   913      ...

   914      >>> transpose([(1,2,3), (4,5,6)])

   915      [(1, 4), (2, 5), (3, 6)]

   916      >>> transpose([])

   917      []

   918 

   919   (Contributed by Raymond Hettinger.)

   920 

   921 * Encountering a failure while importing a module no longer leaves a partially-

   922   initialized module object in ``sys.modules``.  The incomplete module object left

   923   behind would fool further imports of the same module into succeeding, leading to

   924   confusing errors.   (Fixed by Tim Peters.)

   925 

   926 * :const:`None` is now a constant; code that binds a new value to  the name

   927   ``None`` is now a syntax error. (Contributed by Raymond Hettinger.)

   928 

   929 .. ======================================================================

   930 

   931 

   932 Optimizations

   933 -------------

   934 

   935 * The inner loops for list and tuple slicing were optimized and now run about

   936   one-third faster.  The inner loops for dictionaries were also optimized,

   937   resulting in performance boosts for :meth:`keys`, :meth:`values`, :meth:`items`,

   938   :meth:`iterkeys`, :meth:`itervalues`, and :meth:`iteritems`. (Contributed by

   939   Raymond Hettinger.)

   940 

   941 * The machinery for growing and shrinking lists was optimized for speed and for

   942   space efficiency.  Appending and popping from lists now runs faster due to more

   943   efficient code paths and less frequent use of the underlying system

   944   :cfunc:`realloc`.  List comprehensions also benefit.   :meth:`list.extend` was

   945   also optimized and no longer converts its argument into a temporary list before

   946   extending the base list.  (Contributed by Raymond Hettinger.)

   947 

   948 * :func:`list`, :func:`tuple`, :func:`map`, :func:`filter`, and :func:`zip` now

   949   run several times faster with non-sequence arguments that supply a

   950   :meth:`__len__` method.  (Contributed by Raymond Hettinger.)

   951 

   952 * The methods :meth:`list.__getitem__`, :meth:`dict.__getitem__`, and

   953   :meth:`dict.__contains__` are are now implemented as :class:`method_descriptor`

   954   objects rather than :class:`wrapper_descriptor` objects.  This form of  access

   955   doubles their performance and makes them more suitable for use as arguments to

   956   functionals: ``map(mydict.__getitem__, keylist)``. (Contributed by Raymond

   957   Hettinger.)

   958 

   959 * Added a new opcode, ``LIST_APPEND``, that simplifies the generated bytecode

   960   for list comprehensions and speeds them up by about a third.  (Contributed by

   961   Raymond Hettinger.)

   962 

   963 * The peephole bytecode optimizer has been improved to  produce shorter, faster

   964   bytecode; remarkably, the resulting bytecode is  more readable.  (Enhanced by

   965   Raymond Hettinger.)

   966 

   967 * String concatenations in statements of the form ``s = s + "abc"`` and ``s +=

   968   "abc"`` are now performed more efficiently in certain circumstances.  This

   969   optimization won't be present in other Python implementations such as Jython, so

   970   you shouldn't rely on it; using the :meth:`join` method of strings is still

   971   recommended when you want to efficiently glue a large number of strings

   972   together. (Contributed by Armin Rigo.)

   973 

   974 The net result of the 2.4 optimizations is that Python 2.4 runs the pystone

   975 benchmark around 5% faster than Python 2.3 and 35% faster than Python 2.2.

   976 (pystone is not a particularly good benchmark, but it's the most commonly used

   977 measurement of Python's performance.  Your own applications may show greater or

   978 smaller benefits from Python 2.4.)

   979 

   980 .. pystone is almost useless for comparing different versions of Python;

   981    instead, it excels at predicting relative Python performance on different

   982    machines.  So, this section would be more informative if it used other tools

   983    such as pybench and parrotbench.  For a more application oriented benchmark,

   984    try comparing the timings of test_decimal.py under 2.3 and 2.4.

   985 

   986 .. ======================================================================

   987 

   988 

   989 New, Improved, and Deprecated Modules

   990 =====================================

   991 

   992 As usual, Python's standard library received a number of enhancements and bug

   993 fixes.  Here's a partial list of the most notable changes, sorted alphabetically

   994 by module name. Consult the :file:`Misc/NEWS` file in the source tree for a more

   995 complete list of changes, or look through the CVS logs for all the details.

   996 

   997 * The :mod:`asyncore` module's :func:`loop` function now has a *count* parameter

   998   that lets you perform a limited number of passes through the polling loop.  The

   999   default is still to loop forever.

  1000 

  1001 * The :mod:`base64` module now has more complete RFC 3548 support for Base64,

  1002   Base32, and Base16 encoding and decoding, including optional case folding and

  1003   optional alternative alphabets. (Contributed by Barry Warsaw.)

  1004 

  1005 * The :mod:`bisect` module now has an underlying C implementation for improved

  1006   performance. (Contributed by Dmitry Vasiliev.)

  1007 

  1008 * The CJKCodecs collections of East Asian codecs, maintained by Hye-Shik Chang,

  1009   was integrated into 2.4.   The new encodings are:

  1010 

  1011 * Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz

  1012 

  1013 * Chinese (ROC): big5, cp950

  1014 

  1015 * Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp,

  1016     iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext, iso-2022-jp-2004,

  1017     shift-jis, shift-jisx0213, shift-jis-2004

  1018 

  1019 * Korean: cp949, euc-kr, johab, iso-2022-kr

  1020 

  1021 * Some other new encodings were added: HP Roman8,  ISO_8859-11, ISO_8859-16,

  1022   PCTP-154, and TIS-620.

  1023 

  1024 * The UTF-8 and UTF-16 codecs now cope better with receiving partial input.

  1025   Previously the :class:`StreamReader` class would try to read more data, making

  1026   it impossible to resume decoding from the stream.  The :meth:`read` method will

  1027   now return as much data as it can and future calls will resume decoding where

  1028   previous ones left off.  (Implemented by Walter Dörwald.)

  1029 

  1030 * There is a new :mod:`collections` module for  various specialized collection

  1031   datatypes.   Currently it contains just one type, :class:`deque`,  a double-

  1032   ended queue that supports efficiently adding and removing elements from either

  1033   end::

  1034 

  1035      >>> from collections import deque

  1036      >>> d = deque('ghi')        # make a new deque with three items

  1037      >>> d.append('j')           # add a new entry to the right side

  1038      >>> d.appendleft('f')       # add a new entry to the left side

  1039      >>> d                       # show the representation of the deque

  1040      deque(['f', 'g', 'h', 'i', 'j'])

  1041      >>> d.pop()                 # return and remove the rightmost item

  1042      'j'

  1043      >>> d.popleft()             # return and remove the leftmost item

  1044      'f'

  1045      >>> list(d)                 # list the contents of the deque

  1046      ['g', 'h', 'i']

  1047      >>> 'h' in d                # search the deque

  1048      True  

  1049 

  1050   Several modules, such as the :mod:`Queue` and :mod:`threading` modules, now take

  1051   advantage of :class:`collections.deque` for improved performance.  (Contributed

  1052   by Raymond Hettinger.)

  1053 

  1054 * The :mod:`ConfigParser` classes have been enhanced slightly. The :meth:`read`

  1055   method now returns a list of the files that were successfully parsed, and the

  1056   :meth:`set` method raises :exc:`TypeError` if passed a *value* argument that

  1057   isn't a string.   (Contributed by John Belmonte and David Goodger.)

  1058 

  1059 * The :mod:`curses` module now supports the ncurses extension

  1060   :func:`use_default_colors`.  On platforms where the terminal supports

  1061   transparency, this makes it possible to use a transparent background.

  1062   (Contributed by Jörg Lehmann.)

  1063 

  1064 * The :mod:`difflib` module now includes an :class:`HtmlDiff` class that creates

  1065   an HTML table showing a side by side comparison of two versions of a text.

  1066   (Contributed by Dan Gass.)

  1067 

  1068 * The :mod:`email` package was updated to version 3.0,  which dropped various

  1069   deprecated APIs and removes support for Python versions earlier than 2.3.  The

  1070   3.0 version of the package uses a new incremental parser for MIME messages,

  1071   available in the :mod:`email.FeedParser` module.  The new parser doesn't require

  1072   reading the entire message into memory, and doesn't throw exceptions if a

  1073   message is malformed; instead it records any problems in the  :attr:`defect`

  1074   attribute of the message.  (Developed by Anthony Baxter, Barry Warsaw, Thomas

  1075   Wouters, and others.)

  1076 

  1077 * The :mod:`heapq` module has been converted to C.  The resulting tenfold

  1078   improvement in speed makes the module suitable for handling high volumes of

  1079   data.  In addition, the module has two new functions :func:`nlargest` and

  1080   :func:`nsmallest` that use heaps to find the N largest or smallest values in a

  1081   dataset without the expense of a full sort.  (Contributed by Raymond Hettinger.)

  1082 

  1083 * The :mod:`httplib` module now contains constants for HTTP status codes defined

  1084   in various HTTP-related RFC documents.  Constants have names such as

  1085   :const:`OK`, :const:`CREATED`, :const:`CONTINUE`, and

  1086   :const:`MOVED_PERMANENTLY`; use pydoc to get a full list.  (Contributed by

  1087   Andrew Eland.)

  1088 

  1089 * The :mod:`imaplib` module now supports IMAP's THREAD command (contributed by

  1090   Yves Dionne) and new :meth:`deleteacl` and :meth:`myrights` methods (contributed

  1091   by Arnaud Mazin).

  1092 

  1093 * The :mod:`itertools` module gained a :func:`groupby(iterable[, *func*])`

  1094   function. *iterable* is something that can be iterated over to return a stream

  1095   of elements, and the optional *func* parameter is a function that takes an

  1096   element and returns a key value; if omitted, the key is simply the element

  1097   itself.  :func:`groupby` then groups the elements into subsequences which have

  1098   matching values of the key, and returns a series of 2-tuples containing the key

  1099   value and an iterator over the subsequence.

  1100 

  1101   Here's an example to make this clearer.  The *key* function simply returns

  1102   whether a number is even or odd, so the result of :func:`groupby` is to return

  1103   consecutive runs of odd or even numbers. ::

  1104 

  1105      >>> import itertools

  1106      >>> L = [2, 4, 6, 7, 8, 9, 11, 12, 14]

  1107      >>> for key_val, it in itertools.groupby(L, lambda x: x % 2):

  1108      ...    print key_val, list(it)

  1109      ... 

  1110      0 [2, 4, 6]

  1111      1 [7]

  1112      0 [8]

  1113      1 [9, 11]

  1114      0 [12, 14]

  1115      >>> 

  1116 

  1117   :func:`groupby` is typically used with sorted input.  The logic for

  1118   :func:`groupby` is similar to the Unix ``uniq`` filter which makes it handy for

  1119   eliminating, counting, or identifying duplicate elements::

  1120 

  1121      >>> word = 'abracadabra'

  1122      >>> letters = sorted(word)   # Turn string into a sorted list of letters

  1123      >>> letters 

  1124      ['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']

  1125      >>> for k, g in itertools.groupby(letters):

  1126      ...    print k, list(g)

  1127      ... 

  1128      a ['a', 'a', 'a', 'a', 'a']

  1129      b ['b', 'b']

  1130      c ['c']

  1131      d ['d']

  1132      r ['r', 'r']

  1133      >>> # List unique letters

  1134      >>> [k for k, g in groupby(letters)]                     

  1135      ['a', 'b', 'c', 'd', 'r']