symbian-qemu-0.9.1-12/python-2.6.1/Doc/extending/extending.rst
changeset 1 2fb8b9db1c86
equal deleted inserted replaced
0:ffa851df0825 1:2fb8b9db1c86
       
     1 .. highlightlang:: c
       
     2 
       
     3 
       
     4 .. _extending-intro:
       
     5 
       
     6 ******************************
       
     7 Extending Python with C or C++
       
     8 ******************************
       
     9 
       
    10 It is quite easy to add new built-in modules to Python, if you know how to
       
    11 program in C.  Such :dfn:`extension modules` can do two things that can't be
       
    12 done directly in Python: they can implement new built-in object types, and they
       
    13 can call C library functions and system calls.
       
    14 
       
    15 To support extensions, the Python API (Application Programmers Interface)
       
    16 defines a set of functions, macros and variables that provide access to most
       
    17 aspects of the Python run-time system.  The Python API is incorporated in a C
       
    18 source file by including the header ``"Python.h"``.
       
    19 
       
    20 The compilation of an extension module depends on its intended use as well as on
       
    21 your system setup; details are given in later chapters.
       
    22 
       
    23 
       
    24 .. _extending-simpleexample:
       
    25 
       
    26 A Simple Example
       
    27 ================
       
    28 
       
    29 Let's create an extension module called ``spam`` (the favorite food of Monty
       
    30 Python fans...) and let's say we want to create a Python interface to the C
       
    31 library function :cfunc:`system`. [#]_ This function takes a null-terminated
       
    32 character string as argument and returns an integer.  We want this function to
       
    33 be callable from Python as follows::
       
    34 
       
    35    >>> import spam
       
    36    >>> status = spam.system("ls -l")
       
    37 
       
    38 Begin by creating a file :file:`spammodule.c`.  (Historically, if a module is
       
    39 called ``spam``, the C file containing its implementation is called
       
    40 :file:`spammodule.c`; if the module name is very long, like ``spammify``, the
       
    41 module name can be just :file:`spammify.c`.)
       
    42 
       
    43 The first line of our file can be::
       
    44 
       
    45    #include <Python.h>
       
    46 
       
    47 which pulls in the Python API (you can add a comment describing the purpose of
       
    48 the module and a copyright notice if you like).
       
    49 
       
    50 .. warning::
       
    51 
       
    52    Since Python may define some pre-processor definitions which affect the standard
       
    53    headers on some systems, you *must* include :file:`Python.h` before any standard
       
    54    headers are included.
       
    55 
       
    56 All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or
       
    57 ``PY``, except those defined in standard header files. For convenience, and
       
    58 since they are used extensively by the Python interpreter, ``"Python.h"``
       
    59 includes a few standard header files: ``<stdio.h>``, ``<string.h>``,
       
    60 ``<errno.h>``, and ``<stdlib.h>``.  If the latter header file does not exist on
       
    61 your system, it declares the functions :cfunc:`malloc`, :cfunc:`free` and
       
    62 :cfunc:`realloc` directly.
       
    63 
       
    64 The next thing we add to our module file is the C function that will be called
       
    65 when the Python expression ``spam.system(string)`` is evaluated (we'll see
       
    66 shortly how it ends up being called)::
       
    67 
       
    68    static PyObject *
       
    69    spam_system(PyObject *self, PyObject *args)
       
    70    {
       
    71        const char *command;
       
    72        int sts;
       
    73 
       
    74        if (!PyArg_ParseTuple(args, "s", &command))
       
    75            return NULL;
       
    76        sts = system(command);
       
    77        return Py_BuildValue("i", sts);
       
    78    }
       
    79 
       
    80 There is a straightforward translation from the argument list in Python (for
       
    81 example, the single expression ``"ls -l"``) to the arguments passed to the C
       
    82 function.  The C function always has two arguments, conventionally named *self*
       
    83 and *args*.
       
    84 
       
    85 The *self* argument is only used when the C function implements a built-in
       
    86 method, not a function. In the example, *self* will always be a *NULL* pointer,
       
    87 since we are defining a function, not a method.  (This is done so that the
       
    88 interpreter doesn't have to understand two different types of C functions.)
       
    89 
       
    90 The *args* argument will be a pointer to a Python tuple object containing the
       
    91 arguments.  Each item of the tuple corresponds to an argument in the call's
       
    92 argument list.  The arguments are Python objects --- in order to do anything
       
    93 with them in our C function we have to convert them to C values.  The function
       
    94 :cfunc:`PyArg_ParseTuple` in the Python API checks the argument types and
       
    95 converts them to C values.  It uses a template string to determine the required
       
    96 types of the arguments as well as the types of the C variables into which to
       
    97 store the converted values.  More about this later.
       
    98 
       
    99 :cfunc:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right
       
   100 type and its components have been stored in the variables whose addresses are
       
   101 passed.  It returns false (zero) if an invalid argument list was passed.  In the
       
   102 latter case it also raises an appropriate exception so the calling function can
       
   103 return *NULL* immediately (as we saw in the example).
       
   104 
       
   105 
       
   106 .. _extending-errors:
       
   107 
       
   108 Intermezzo: Errors and Exceptions
       
   109 =================================
       
   110 
       
   111 An important convention throughout the Python interpreter is the following: when
       
   112 a function fails, it should set an exception condition and return an error value
       
   113 (usually a *NULL* pointer).  Exceptions are stored in a static global variable
       
   114 inside the interpreter; if this variable is *NULL* no exception has occurred.  A
       
   115 second global variable stores the "associated value" of the exception (the
       
   116 second argument to :keyword:`raise`).  A third variable contains the stack
       
   117 traceback in case the error originated in Python code.  These three variables
       
   118 are the C equivalents of the Python variables ``sys.exc_type``,
       
   119 ``sys.exc_value`` and ``sys.exc_traceback`` (see the section on module
       
   120 :mod:`sys` in the Python Library Reference).  It is important to know about them
       
   121 to understand how errors are passed around.
       
   122 
       
   123 The Python API defines a number of functions to set various types of exceptions.
       
   124 
       
   125 The most common one is :cfunc:`PyErr_SetString`.  Its arguments are an exception
       
   126 object and a C string.  The exception object is usually a predefined object like
       
   127 :cdata:`PyExc_ZeroDivisionError`.  The C string indicates the cause of the error
       
   128 and is converted to a Python string object and stored as the "associated value"
       
   129 of the exception.
       
   130 
       
   131 Another useful function is :cfunc:`PyErr_SetFromErrno`, which only takes an
       
   132 exception argument and constructs the associated value by inspection of the
       
   133 global variable :cdata:`errno`.  The most general function is
       
   134 :cfunc:`PyErr_SetObject`, which takes two object arguments, the exception and
       
   135 its associated value.  You don't need to :cfunc:`Py_INCREF` the objects passed
       
   136 to any of these functions.
       
   137 
       
   138 You can test non-destructively whether an exception has been set with
       
   139 :cfunc:`PyErr_Occurred`.  This returns the current exception object, or *NULL*
       
   140 if no exception has occurred.  You normally don't need to call
       
   141 :cfunc:`PyErr_Occurred` to see whether an error occurred in a function call,
       
   142 since you should be able to tell from the return value.
       
   143 
       
   144 When a function *f* that calls another function *g* detects that the latter
       
   145 fails, *f* should itself return an error value (usually *NULL* or ``-1``).  It
       
   146 should *not* call one of the :cfunc:`PyErr_\*` functions --- one has already
       
   147 been called by *g*. *f*'s caller is then supposed to also return an error
       
   148 indication to *its* caller, again *without* calling :cfunc:`PyErr_\*`, and so on
       
   149 --- the most detailed cause of the error was already reported by the function
       
   150 that first detected it.  Once the error reaches the Python interpreter's main
       
   151 loop, this aborts the currently executing Python code and tries to find an
       
   152 exception handler specified by the Python programmer.
       
   153 
       
   154 (There are situations where a module can actually give a more detailed error
       
   155 message by calling another :cfunc:`PyErr_\*` function, and in such cases it is
       
   156 fine to do so.  As a general rule, however, this is not necessary, and can cause
       
   157 information about the cause of the error to be lost: most operations can fail
       
   158 for a variety of reasons.)
       
   159 
       
   160 To ignore an exception set by a function call that failed, the exception
       
   161 condition must be cleared explicitly by calling :cfunc:`PyErr_Clear`.  The only
       
   162 time C code should call :cfunc:`PyErr_Clear` is if it doesn't want to pass the
       
   163 error on to the interpreter but wants to handle it completely by itself
       
   164 (possibly by trying something else, or pretending nothing went wrong).
       
   165 
       
   166 Every failing :cfunc:`malloc` call must be turned into an exception --- the
       
   167 direct caller of :cfunc:`malloc` (or :cfunc:`realloc`) must call
       
   168 :cfunc:`PyErr_NoMemory` and return a failure indicator itself.  All the
       
   169 object-creating functions (for example, :cfunc:`PyInt_FromLong`) already do
       
   170 this, so this note is only relevant to those who call :cfunc:`malloc` directly.
       
   171 
       
   172 Also note that, with the important exception of :cfunc:`PyArg_ParseTuple` and
       
   173 friends, functions that return an integer status usually return a positive value
       
   174 or zero for success and ``-1`` for failure, like Unix system calls.
       
   175 
       
   176 Finally, be careful to clean up garbage (by making :cfunc:`Py_XDECREF` or
       
   177 :cfunc:`Py_DECREF` calls for objects you have already created) when you return
       
   178 an error indicator!
       
   179 
       
   180 The choice of which exception to raise is entirely yours.  There are predeclared
       
   181 C objects corresponding to all built-in Python exceptions, such as
       
   182 :cdata:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you
       
   183 should choose exceptions wisely --- don't use :cdata:`PyExc_TypeError` to mean
       
   184 that a file couldn't be opened (that should probably be :cdata:`PyExc_IOError`).
       
   185 If something's wrong with the argument list, the :cfunc:`PyArg_ParseTuple`
       
   186 function usually raises :cdata:`PyExc_TypeError`.  If you have an argument whose
       
   187 value must be in a particular range or must satisfy other conditions,
       
   188 :cdata:`PyExc_ValueError` is appropriate.
       
   189 
       
   190 You can also define a new exception that is unique to your module. For this, you
       
   191 usually declare a static object variable at the beginning of your file::
       
   192 
       
   193    static PyObject *SpamError;
       
   194 
       
   195 and initialize it in your module's initialization function (:cfunc:`initspam`)
       
   196 with an exception object (leaving out the error checking for now)::
       
   197 
       
   198    PyMODINIT_FUNC
       
   199    initspam(void)
       
   200    {
       
   201        PyObject *m;
       
   202 
       
   203        m = Py_InitModule("spam", SpamMethods);
       
   204        if (m == NULL)
       
   205            return;
       
   206 
       
   207        SpamError = PyErr_NewException("spam.error", NULL, NULL);
       
   208        Py_INCREF(SpamError);
       
   209        PyModule_AddObject(m, "error", SpamError);
       
   210    }
       
   211 
       
   212 Note that the Python name for the exception object is :exc:`spam.error`.  The
       
   213 :cfunc:`PyErr_NewException` function may create a class with the base class
       
   214 being :exc:`Exception` (unless another class is passed in instead of *NULL*),
       
   215 described in :ref:`bltin-exceptions`.
       
   216 
       
   217 Note also that the :cdata:`SpamError` variable retains a reference to the newly
       
   218 created exception class; this is intentional!  Since the exception could be
       
   219 removed from the module by external code, an owned reference to the class is
       
   220 needed to ensure that it will not be discarded, causing :cdata:`SpamError` to
       
   221 become a dangling pointer. Should it become a dangling pointer, C code which
       
   222 raises the exception could cause a core dump or other unintended side effects.
       
   223 
       
   224 We discuss the use of PyMODINIT_FUNC as a function return type later in this
       
   225 sample.
       
   226 
       
   227 
       
   228 .. _backtoexample:
       
   229 
       
   230 Back to the Example
       
   231 ===================
       
   232 
       
   233 Going back to our example function, you should now be able to understand this
       
   234 statement::
       
   235 
       
   236    if (!PyArg_ParseTuple(args, "s", &command))
       
   237        return NULL;
       
   238 
       
   239 It returns *NULL* (the error indicator for functions returning object pointers)
       
   240 if an error is detected in the argument list, relying on the exception set by
       
   241 :cfunc:`PyArg_ParseTuple`.  Otherwise the string value of the argument has been
       
   242 copied to the local variable :cdata:`command`.  This is a pointer assignment and
       
   243 you are not supposed to modify the string to which it points (so in Standard C,
       
   244 the variable :cdata:`command` should properly be declared as ``const char
       
   245 *command``).
       
   246 
       
   247 The next statement is a call to the Unix function :cfunc:`system`, passing it
       
   248 the string we just got from :cfunc:`PyArg_ParseTuple`::
       
   249 
       
   250    sts = system(command);
       
   251 
       
   252 Our :func:`spam.system` function must return the value of :cdata:`sts` as a
       
   253 Python object.  This is done using the function :cfunc:`Py_BuildValue`, which is
       
   254 something like the inverse of :cfunc:`PyArg_ParseTuple`: it takes a format
       
   255 string and an arbitrary number of C values, and returns a new Python object.
       
   256 More info on :cfunc:`Py_BuildValue` is given later. ::
       
   257 
       
   258    return Py_BuildValue("i", sts);
       
   259 
       
   260 In this case, it will return an integer object.  (Yes, even integers are objects
       
   261 on the heap in Python!)
       
   262 
       
   263 If you have a C function that returns no useful argument (a function returning
       
   264 :ctype:`void`), the corresponding Python function must return ``None``.   You
       
   265 need this idiom to do so (which is implemented by the :cmacro:`Py_RETURN_NONE`
       
   266 macro)::
       
   267 
       
   268    Py_INCREF(Py_None);
       
   269    return Py_None;
       
   270 
       
   271 :cdata:`Py_None` is the C name for the special Python object ``None``.  It is a
       
   272 genuine Python object rather than a *NULL* pointer, which means "error" in most
       
   273 contexts, as we have seen.
       
   274 
       
   275 
       
   276 .. _methodtable:
       
   277 
       
   278 The Module's Method Table and Initialization Function
       
   279 =====================================================
       
   280 
       
   281 I promised to show how :cfunc:`spam_system` is called from Python programs.
       
   282 First, we need to list its name and address in a "method table"::
       
   283 
       
   284    static PyMethodDef SpamMethods[] = {
       
   285        ...
       
   286        {"system",  spam_system, METH_VARARGS,
       
   287         "Execute a shell command."},
       
   288        ...
       
   289        {NULL, NULL, 0, NULL}        /* Sentinel */
       
   290    };
       
   291 
       
   292 Note the third entry (``METH_VARARGS``).  This is a flag telling the interpreter
       
   293 the calling convention to be used for the C function.  It should normally always
       
   294 be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means
       
   295 that an obsolete variant of :cfunc:`PyArg_ParseTuple` is used.
       
   296 
       
   297 When using only ``METH_VARARGS``, the function should expect the Python-level
       
   298 parameters to be passed in as a tuple acceptable for parsing via
       
   299 :cfunc:`PyArg_ParseTuple`; more information on this function is provided below.
       
   300 
       
   301 The :const:`METH_KEYWORDS` bit may be set in the third field if keyword
       
   302 arguments should be passed to the function.  In this case, the C function should
       
   303 accept a third ``PyObject *`` parameter which will be a dictionary of keywords.
       
   304 Use :cfunc:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a
       
   305 function.
       
   306 
       
   307 The method table must be passed to the interpreter in the module's
       
   308 initialization function.  The initialization function must be named
       
   309 :cfunc:`initname`, where *name* is the name of the module, and should be the
       
   310 only non-\ ``static`` item defined in the module file::
       
   311 
       
   312    PyMODINIT_FUNC
       
   313    initspam(void)
       
   314    {
       
   315        (void) Py_InitModule("spam", SpamMethods);
       
   316    }
       
   317 
       
   318 Note that PyMODINIT_FUNC declares the function as ``void`` return type,
       
   319 declares any special linkage declarations required by the platform, and for  C++
       
   320 declares the function as ``extern "C"``.
       
   321 
       
   322 When the Python program imports module :mod:`spam` for the first time,
       
   323 :cfunc:`initspam` is called. (See below for comments about embedding Python.)
       
   324 It calls :cfunc:`Py_InitModule`, which creates a "module object" (which is
       
   325 inserted in the dictionary ``sys.modules`` under the key ``"spam"``), and
       
   326 inserts built-in function objects into the newly created module based upon the
       
   327 table (an array of :ctype:`PyMethodDef` structures) that was passed as its
       
   328 second argument. :cfunc:`Py_InitModule` returns a pointer to the module object
       
   329 that it creates (which is unused here).  It may abort with a fatal error for
       
   330 certain errors, or return *NULL* if the module could not be initialized
       
   331 satisfactorily.
       
   332 
       
   333 When embedding Python, the :cfunc:`initspam` function is not called
       
   334 automatically unless there's an entry in the :cdata:`_PyImport_Inittab` table.
       
   335 The easiest way to handle this is to statically initialize your
       
   336 statically-linked modules by directly calling :cfunc:`initspam` after the call
       
   337 to :cfunc:`Py_Initialize`::
       
   338 
       
   339    int
       
   340    main(int argc, char *argv[])
       
   341    {
       
   342        /* Pass argv[0] to the Python interpreter */
       
   343        Py_SetProgramName(argv[0]);
       
   344 
       
   345        /* Initialize the Python interpreter.  Required. */
       
   346        Py_Initialize();
       
   347 
       
   348        /* Add a static module */
       
   349        initspam();
       
   350 
       
   351 An example may be found in the file :file:`Demo/embed/demo.c` in the Python
       
   352 source distribution.
       
   353 
       
   354 .. note::
       
   355 
       
   356    Removing entries from ``sys.modules`` or importing compiled modules into
       
   357    multiple interpreters within a process (or following a :cfunc:`fork` without an
       
   358    intervening :cfunc:`exec`) can create problems for some extension modules.
       
   359    Extension module authors should exercise caution when initializing internal data
       
   360    structures. Note also that the :func:`reload` function can be used with
       
   361    extension modules, and will call the module initialization function
       
   362    (:cfunc:`initspam` in the example), but will not load the module again if it was
       
   363    loaded from a dynamically loadable object file (:file:`.so` on Unix,
       
   364    :file:`.dll` on Windows).
       
   365 
       
   366 A more substantial example module is included in the Python source distribution
       
   367 as :file:`Modules/xxmodule.c`.  This file may be used as a  template or simply
       
   368 read as an example.  The :program:`modulator.py` script included in the source
       
   369 distribution or Windows install provides  a simple graphical user interface for
       
   370 declaring the functions and objects which a module should implement, and can
       
   371 generate a template which can be filled in.  The script lives in the
       
   372 :file:`Tools/modulator/` directory; see the :file:`README` file there for more
       
   373 information.
       
   374 
       
   375 
       
   376 .. _compilation:
       
   377 
       
   378 Compilation and Linkage
       
   379 =======================
       
   380 
       
   381 There are two more things to do before you can use your new extension: compiling
       
   382 and linking it with the Python system.  If you use dynamic loading, the details
       
   383 may depend on the style of dynamic loading your system uses; see the chapters
       
   384 about building extension modules (chapter :ref:`building`) and additional
       
   385 information that pertains only to building on Windows (chapter
       
   386 :ref:`building-on-windows`) for more information about this.
       
   387 
       
   388 If you can't use dynamic loading, or if you want to make your module a permanent
       
   389 part of the Python interpreter, you will have to change the configuration setup
       
   390 and rebuild the interpreter.  Luckily, this is very simple on Unix: just place
       
   391 your file (:file:`spammodule.c` for example) in the :file:`Modules/` directory
       
   392 of an unpacked source distribution, add a line to the file
       
   393 :file:`Modules/Setup.local` describing your file::
       
   394 
       
   395    spam spammodule.o
       
   396 
       
   397 and rebuild the interpreter by running :program:`make` in the toplevel
       
   398 directory.  You can also run :program:`make` in the :file:`Modules/`
       
   399 subdirectory, but then you must first rebuild :file:`Makefile` there by running
       
   400 ':program:`make` Makefile'.  (This is necessary each time you change the
       
   401 :file:`Setup` file.)
       
   402 
       
   403 If your module requires additional libraries to link with, these can be listed
       
   404 on the line in the configuration file as well, for instance::
       
   405 
       
   406    spam spammodule.o -lX11
       
   407 
       
   408 
       
   409 .. _callingpython:
       
   410 
       
   411 Calling Python Functions from C
       
   412 ===============================
       
   413 
       
   414 So far we have concentrated on making C functions callable from Python.  The
       
   415 reverse is also useful: calling Python functions from C. This is especially the
       
   416 case for libraries that support so-called "callback" functions.  If a C
       
   417 interface makes use of callbacks, the equivalent Python often needs to provide a
       
   418 callback mechanism to the Python programmer; the implementation will require
       
   419 calling the Python callback functions from a C callback.  Other uses are also
       
   420 imaginable.
       
   421 
       
   422 Fortunately, the Python interpreter is easily called recursively, and there is a
       
   423 standard interface to call a Python function.  (I won't dwell on how to call the
       
   424 Python parser with a particular string as input --- if you're interested, have a
       
   425 look at the implementation of the :option:`-c` command line option in
       
   426 :file:`Modules/main.c` from the Python source code.)
       
   427 
       
   428 Calling a Python function is easy.  First, the Python program must somehow pass
       
   429 you the Python function object.  You should provide a function (or some other
       
   430 interface) to do this.  When this function is called, save a pointer to the
       
   431 Python function object (be careful to :cfunc:`Py_INCREF` it!) in a global
       
   432 variable --- or wherever you see fit. For example, the following function might
       
   433 be part of a module definition::
       
   434 
       
   435    static PyObject *my_callback = NULL;
       
   436 
       
   437    static PyObject *
       
   438    my_set_callback(PyObject *dummy, PyObject *args)
       
   439    {
       
   440        PyObject *result = NULL;
       
   441        PyObject *temp;
       
   442 
       
   443        if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
       
   444            if (!PyCallable_Check(temp)) {
       
   445                PyErr_SetString(PyExc_TypeError, "parameter must be callable");
       
   446                return NULL;
       
   447            }
       
   448            Py_XINCREF(temp);         /* Add a reference to new callback */
       
   449            Py_XDECREF(my_callback);  /* Dispose of previous callback */
       
   450            my_callback = temp;       /* Remember new callback */
       
   451            /* Boilerplate to return "None" */
       
   452            Py_INCREF(Py_None);
       
   453            result = Py_None;
       
   454        }
       
   455        return result;
       
   456    }
       
   457 
       
   458 This function must be registered with the interpreter using the
       
   459 :const:`METH_VARARGS` flag; this is described in section :ref:`methodtable`.  The
       
   460 :cfunc:`PyArg_ParseTuple` function and its arguments are documented in section
       
   461 :ref:`parsetuple`.
       
   462 
       
   463 The macros :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` increment/decrement the
       
   464 reference count of an object and are safe in the presence of *NULL* pointers
       
   465 (but note that *temp* will not be  *NULL* in this context).  More info on them
       
   466 in section :ref:`refcounts`.
       
   467 
       
   468 .. index:: single: PyEval_CallObject()
       
   469 
       
   470 Later, when it is time to call the function, you call the C function
       
   471 :cfunc:`PyEval_CallObject`.  This function has two arguments, both pointers to
       
   472 arbitrary Python objects: the Python function, and the argument list.  The
       
   473 argument list must always be a tuple object, whose length is the number of
       
   474 arguments.  To call the Python function with no arguments, pass in NULL, or 
       
   475 an empty tuple; to call it with one argument, pass a singleton tuple.
       
   476 :cfunc:`Py_BuildValue` returns a tuple when its format string consists of zero
       
   477 or more format codes between parentheses.  For example::
       
   478 
       
   479    int arg;
       
   480    PyObject *arglist;
       
   481    PyObject *result;
       
   482    ...
       
   483    arg = 123;
       
   484    ...
       
   485    /* Time to call the callback */
       
   486    arglist = Py_BuildValue("(i)", arg);
       
   487    result = PyEval_CallObject(my_callback, arglist);
       
   488    Py_DECREF(arglist);
       
   489 
       
   490 :cfunc:`PyEval_CallObject` returns a Python object pointer: this is the return
       
   491 value of the Python function.  :cfunc:`PyEval_CallObject` is
       
   492 "reference-count-neutral" with respect to its arguments.  In the example a new
       
   493 tuple was created to serve as the argument list, which is :cfunc:`Py_DECREF`\
       
   494 -ed immediately after the call.
       
   495 
       
   496 The return value of :cfunc:`PyEval_CallObject` is "new": either it is a brand
       
   497 new object, or it is an existing object whose reference count has been
       
   498 incremented.  So, unless you want to save it in a global variable, you should
       
   499 somehow :cfunc:`Py_DECREF` the result, even (especially!) if you are not
       
   500 interested in its value.
       
   501 
       
   502 Before you do this, however, it is important to check that the return value
       
   503 isn't *NULL*.  If it is, the Python function terminated by raising an exception.
       
   504 If the C code that called :cfunc:`PyEval_CallObject` is called from Python, it
       
   505 should now return an error indication to its Python caller, so the interpreter
       
   506 can print a stack trace, or the calling Python code can handle the exception.
       
   507 If this is not possible or desirable, the exception should be cleared by calling
       
   508 :cfunc:`PyErr_Clear`.  For example::
       
   509 
       
   510    if (result == NULL)
       
   511        return NULL; /* Pass error back */
       
   512    ...use result...
       
   513    Py_DECREF(result); 
       
   514 
       
   515 Depending on the desired interface to the Python callback function, you may also
       
   516 have to provide an argument list to :cfunc:`PyEval_CallObject`.  In some cases
       
   517 the argument list is also provided by the Python program, through the same
       
   518 interface that specified the callback function.  It can then be saved and used
       
   519 in the same manner as the function object.  In other cases, you may have to
       
   520 construct a new tuple to pass as the argument list.  The simplest way to do this
       
   521 is to call :cfunc:`Py_BuildValue`.  For example, if you want to pass an integral
       
   522 event code, you might use the following code::
       
   523 
       
   524    PyObject *arglist;
       
   525    ...
       
   526    arglist = Py_BuildValue("(l)", eventcode);
       
   527    result = PyEval_CallObject(my_callback, arglist);
       
   528    Py_DECREF(arglist);
       
   529    if (result == NULL)
       
   530        return NULL; /* Pass error back */
       
   531    /* Here maybe use the result */
       
   532    Py_DECREF(result);
       
   533 
       
   534 Note the placement of ``Py_DECREF(arglist)`` immediately after the call, before
       
   535 the error check!  Also note that strictly speaking this code is not complete:
       
   536 :cfunc:`Py_BuildValue` may run out of memory, and this should be checked.
       
   537 
       
   538 You may also call a function with keyword arguments by using 
       
   539 :cfunc:`PyEval_CallObjectWithKeywords`.  As in the above example, we use
       
   540 :cfunc:`Py_BuildValue` to construct the dictionary. ::
       
   541 
       
   542    PyObject *dict;
       
   543    ...
       
   544    dict = Py_BuildValue("{s:i}", "name", val);
       
   545    result = PyEval_CallObjectWithKeywords(my_callback, NULL, dict);
       
   546    Py_DECREF(dict);
       
   547    if (result == NULL)
       
   548        return NULL; /* Pass error back */
       
   549    /* Here maybe use the result */
       
   550    Py_DECREF(result);
       
   551 
       
   552 .. _parsetuple:
       
   553 
       
   554 Extracting Parameters in Extension Functions
       
   555 ============================================
       
   556 
       
   557 .. index:: single: PyArg_ParseTuple()
       
   558 
       
   559 The :cfunc:`PyArg_ParseTuple` function is declared as follows::
       
   560 
       
   561    int PyArg_ParseTuple(PyObject *arg, char *format, ...);
       
   562 
       
   563 The *arg* argument must be a tuple object containing an argument list passed
       
   564 from Python to a C function.  The *format* argument must be a format string,
       
   565 whose syntax is explained in :ref:`arg-parsing` in the Python/C API Reference
       
   566 Manual.  The remaining arguments must be addresses of variables whose type is
       
   567 determined by the format string.
       
   568 
       
   569 Note that while :cfunc:`PyArg_ParseTuple` checks that the Python arguments have
       
   570 the required types, it cannot check the validity of the addresses of C variables
       
   571 passed to the call: if you make mistakes there, your code will probably crash or
       
   572 at least overwrite random bits in memory.  So be careful!
       
   573 
       
   574 Note that any Python object references which are provided to the caller are
       
   575 *borrowed* references; do not decrement their reference count!
       
   576 
       
   577 Some example calls::
       
   578 
       
   579    int ok;
       
   580    int i, j;
       
   581    long k, l;
       
   582    const char *s;
       
   583    int size;
       
   584 
       
   585    ok = PyArg_ParseTuple(args, ""); /* No arguments */
       
   586        /* Python call: f() */
       
   587 
       
   588 ::
       
   589 
       
   590    ok = PyArg_ParseTuple(args, "s", &s); /* A string */
       
   591        /* Possible Python call: f('whoops!') */
       
   592 
       
   593 ::
       
   594 
       
   595    ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
       
   596        /* Possible Python call: f(1, 2, 'three') */
       
   597 
       
   598 ::
       
   599 
       
   600    ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
       
   601        /* A pair of ints and a string, whose size is also returned */
       
   602        /* Possible Python call: f((1, 2), 'three') */
       
   603 
       
   604 ::
       
   605 
       
   606    {
       
   607        const char *file;
       
   608        const char *mode = "r";
       
   609        int bufsize = 0;
       
   610        ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
       
   611        /* A string, and optionally another string and an integer */
       
   612        /* Possible Python calls:
       
   613           f('spam')
       
   614           f('spam', 'w')
       
   615           f('spam', 'wb', 100000) */
       
   616    }
       
   617 
       
   618 ::
       
   619 
       
   620    {
       
   621        int left, top, right, bottom, h, v;
       
   622        ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
       
   623                 &left, &top, &right, &bottom, &h, &v);
       
   624        /* A rectangle and a point */
       
   625        /* Possible Python call:
       
   626           f(((0, 0), (400, 300)), (10, 10)) */
       
   627    }
       
   628 
       
   629 ::
       
   630 
       
   631    {
       
   632        Py_complex c;
       
   633        ok = PyArg_ParseTuple(args, "D:myfunction", &c);
       
   634        /* a complex, also providing a function name for errors */
       
   635        /* Possible Python call: myfunction(1+2j) */
       
   636    }
       
   637 
       
   638 
       
   639 .. _parsetupleandkeywords:
       
   640 
       
   641 Keyword Parameters for Extension Functions
       
   642 ==========================================
       
   643 
       
   644 .. index:: single: PyArg_ParseTupleAndKeywords()
       
   645 
       
   646 The :cfunc:`PyArg_ParseTupleAndKeywords` function is declared as follows::
       
   647 
       
   648    int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
       
   649                                    char *format, char *kwlist[], ...);
       
   650 
       
   651 The *arg* and *format* parameters are identical to those of the
       
   652 :cfunc:`PyArg_ParseTuple` function.  The *kwdict* parameter is the dictionary of
       
   653 keywords received as the third parameter from the Python runtime.  The *kwlist*
       
   654 parameter is a *NULL*-terminated list of strings which identify the parameters;
       
   655 the names are matched with the type information from *format* from left to
       
   656 right.  On success, :cfunc:`PyArg_ParseTupleAndKeywords` returns true, otherwise
       
   657 it returns false and raises an appropriate exception.
       
   658 
       
   659 .. note::
       
   660 
       
   661    Nested tuples cannot be parsed when using keyword arguments!  Keyword parameters
       
   662    passed in which are not present in the *kwlist* will cause :exc:`TypeError` to
       
   663    be raised.
       
   664 
       
   665 .. index:: single: Philbrick, Geoff
       
   666 
       
   667 Here is an example module which uses keywords, based on an example by Geoff
       
   668 Philbrick (philbrick@hks.com)::
       
   669 
       
   670    #include "Python.h"
       
   671 
       
   672    static PyObject *
       
   673    keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
       
   674    {  
       
   675        int voltage;
       
   676        char *state = "a stiff";
       
   677        char *action = "voom";
       
   678        char *type = "Norwegian Blue";
       
   679 
       
   680        static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
       
   681 
       
   682        if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist, 
       
   683                                         &voltage, &state, &action, &type))
       
   684            return NULL; 
       
   685 
       
   686        printf("-- This parrot wouldn't %s if you put %i Volts through it.\n", 
       
   687               action, voltage);
       
   688        printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);
       
   689 
       
   690        Py_INCREF(Py_None);
       
   691 
       
   692        return Py_None;
       
   693    }
       
   694 
       
   695    static PyMethodDef keywdarg_methods[] = {
       
   696        /* The cast of the function is necessary since PyCFunction values
       
   697         * only take two PyObject* parameters, and keywdarg_parrot() takes
       
   698         * three.
       
   699         */
       
   700        {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
       
   701         "Print a lovely skit to standard output."},
       
   702        {NULL, NULL, 0, NULL}   /* sentinel */
       
   703    };
       
   704 
       
   705 ::
       
   706 
       
   707    void
       
   708    initkeywdarg(void)
       
   709    {
       
   710      /* Create the module and add the functions */
       
   711      Py_InitModule("keywdarg", keywdarg_methods);
       
   712    }
       
   713 
       
   714 
       
   715 .. _buildvalue:
       
   716 
       
   717 Building Arbitrary Values
       
   718 =========================
       
   719 
       
   720 This function is the counterpart to :cfunc:`PyArg_ParseTuple`.  It is declared
       
   721 as follows::
       
   722 
       
   723    PyObject *Py_BuildValue(char *format, ...);
       
   724 
       
   725 It recognizes a set of format units similar to the ones recognized by
       
   726 :cfunc:`PyArg_ParseTuple`, but the arguments (which are input to the function,
       
   727 not output) must not be pointers, just values.  It returns a new Python object,
       
   728 suitable for returning from a C function called from Python.
       
   729 
       
   730 One difference with :cfunc:`PyArg_ParseTuple`: while the latter requires its
       
   731 first argument to be a tuple (since Python argument lists are always represented
       
   732 as tuples internally), :cfunc:`Py_BuildValue` does not always build a tuple.  It
       
   733 builds a tuple only if its format string contains two or more format units. If
       
   734 the format string is empty, it returns ``None``; if it contains exactly one
       
   735 format unit, it returns whatever object is described by that format unit.  To
       
   736 force it to return a tuple of size 0 or one, parenthesize the format string.
       
   737 
       
   738 Examples (to the left the call, to the right the resulting Python value)::
       
   739 
       
   740    Py_BuildValue("")                        None
       
   741    Py_BuildValue("i", 123)                  123
       
   742    Py_BuildValue("iii", 123, 456, 789)      (123, 456, 789)
       
   743    Py_BuildValue("s", "hello")              'hello'
       
   744    Py_BuildValue("ss", "hello", "world")    ('hello', 'world')
       
   745    Py_BuildValue("s#", "hello", 4)          'hell'
       
   746    Py_BuildValue("()")                      ()
       
   747    Py_BuildValue("(i)", 123)                (123,)
       
   748    Py_BuildValue("(ii)", 123, 456)          (123, 456)
       
   749    Py_BuildValue("(i,i)", 123, 456)         (123, 456)
       
   750    Py_BuildValue("[i,i]", 123, 456)         [123, 456]
       
   751    Py_BuildValue("{s:i,s:i}",
       
   752                  "abc", 123, "def", 456)    {'abc': 123, 'def': 456}
       
   753    Py_BuildValue("((ii)(ii)) (ii)",
       
   754                  1, 2, 3, 4, 5, 6)          (((1, 2), (3, 4)), (5, 6))
       
   755 
       
   756 
       
   757 .. _refcounts:
       
   758 
       
   759 Reference Counts
       
   760 ================
       
   761 
       
   762 In languages like C or C++, the programmer is responsible for dynamic allocation
       
   763 and deallocation of memory on the heap.  In C, this is done using the functions
       
   764 :cfunc:`malloc` and :cfunc:`free`.  In C++, the operators ``new`` and
       
   765 ``delete`` are used with essentially the same meaning and we'll restrict
       
   766 the following discussion to the C case.
       
   767 
       
   768 Every block of memory allocated with :cfunc:`malloc` should eventually be
       
   769 returned to the pool of available memory by exactly one call to :cfunc:`free`.
       
   770 It is important to call :cfunc:`free` at the right time.  If a block's address
       
   771 is forgotten but :cfunc:`free` is not called for it, the memory it occupies
       
   772 cannot be reused until the program terminates.  This is called a :dfn:`memory
       
   773 leak`.  On the other hand, if a program calls :cfunc:`free` for a block and then
       
   774 continues to use the block, it creates a conflict with re-use of the block
       
   775 through another :cfunc:`malloc` call.  This is called :dfn:`using freed memory`.
       
   776 It has the same bad consequences as referencing uninitialized data --- core
       
   777 dumps, wrong results, mysterious crashes.
       
   778 
       
   779 Common causes of memory leaks are unusual paths through the code.  For instance,
       
   780 a function may allocate a block of memory, do some calculation, and then free
       
   781 the block again.  Now a change in the requirements for the function may add a
       
   782 test to the calculation that detects an error condition and can return
       
   783 prematurely from the function.  It's easy to forget to free the allocated memory
       
   784 block when taking this premature exit, especially when it is added later to the
       
   785 code.  Such leaks, once introduced, often go undetected for a long time: the
       
   786 error exit is taken only in a small fraction of all calls, and most modern
       
   787 machines have plenty of virtual memory, so the leak only becomes apparent in a
       
   788 long-running process that uses the leaking function frequently.  Therefore, it's
       
   789 important to prevent leaks from happening by having a coding convention or
       
   790 strategy that minimizes this kind of errors.
       
   791 
       
   792 Since Python makes heavy use of :cfunc:`malloc` and :cfunc:`free`, it needs a
       
   793 strategy to avoid memory leaks as well as the use of freed memory.  The chosen
       
   794 method is called :dfn:`reference counting`.  The principle is simple: every
       
   795 object contains a counter, which is incremented when a reference to the object
       
   796 is stored somewhere, and which is decremented when a reference to it is deleted.
       
   797 When the counter reaches zero, the last reference to the object has been deleted
       
   798 and the object is freed.
       
   799 
       
   800 An alternative strategy is called :dfn:`automatic garbage collection`.
       
   801 (Sometimes, reference counting is also referred to as a garbage collection
       
   802 strategy, hence my use of "automatic" to distinguish the two.)  The big
       
   803 advantage of automatic garbage collection is that the user doesn't need to call
       
   804 :cfunc:`free` explicitly.  (Another claimed advantage is an improvement in speed
       
   805 or memory usage --- this is no hard fact however.)  The disadvantage is that for
       
   806 C, there is no truly portable automatic garbage collector, while reference
       
   807 counting can be implemented portably (as long as the functions :cfunc:`malloc`
       
   808 and :cfunc:`free` are available --- which the C Standard guarantees). Maybe some
       
   809 day a sufficiently portable automatic garbage collector will be available for C.
       
   810 Until then, we'll have to live with reference counts.
       
   811 
       
   812 While Python uses the traditional reference counting implementation, it also
       
   813 offers a cycle detector that works to detect reference cycles.  This allows
       
   814 applications to not worry about creating direct or indirect circular references;
       
   815 these are the weakness of garbage collection implemented using only reference
       
   816 counting.  Reference cycles consist of objects which contain (possibly indirect)
       
   817 references to themselves, so that each object in the cycle has a reference count
       
   818 which is non-zero.  Typical reference counting implementations are not able to
       
   819 reclaim the memory belonging to any objects in a reference cycle, or referenced
       
   820 from the objects in the cycle, even though there are no further references to
       
   821 the cycle itself.
       
   822 
       
   823 The cycle detector is able to detect garbage cycles and can reclaim them so long
       
   824 as there are no finalizers implemented in Python (:meth:`__del__` methods).
       
   825 When there are such finalizers, the detector exposes the cycles through the
       
   826 :mod:`gc` module (specifically, the
       
   827 ``garbage`` variable in that module).  The :mod:`gc` module also exposes a way
       
   828 to run the detector (the :func:`collect` function), as well as configuration
       
   829 interfaces and the ability to disable the detector at runtime.  The cycle
       
   830 detector is considered an optional component; though it is included by default,
       
   831 it can be disabled at build time using the :option:`--without-cycle-gc` option
       
   832 to the :program:`configure` script on Unix platforms (including Mac OS X) or by
       
   833 removing the definition of ``WITH_CYCLE_GC`` in the :file:`pyconfig.h` header on
       
   834 other platforms.  If the cycle detector is disabled in this way, the :mod:`gc`
       
   835 module will not be available.
       
   836 
       
   837 
       
   838 .. _refcountsinpython:
       
   839 
       
   840 Reference Counting in Python
       
   841 ----------------------------
       
   842 
       
   843 There are two macros, ``Py_INCREF(x)`` and ``Py_DECREF(x)``, which handle the
       
   844 incrementing and decrementing of the reference count. :cfunc:`Py_DECREF` also
       
   845 frees the object when the count reaches zero. For flexibility, it doesn't call
       
   846 :cfunc:`free` directly --- rather, it makes a call through a function pointer in
       
   847 the object's :dfn:`type object`.  For this purpose (and others), every object
       
   848 also contains a pointer to its type object.
       
   849 
       
   850 The big question now remains: when to use ``Py_INCREF(x)`` and ``Py_DECREF(x)``?
       
   851 Let's first introduce some terms.  Nobody "owns" an object; however, you can
       
   852 :dfn:`own a reference` to an object.  An object's reference count is now defined
       
   853 as the number of owned references to it.  The owner of a reference is
       
   854 responsible for calling :cfunc:`Py_DECREF` when the reference is no longer
       
   855 needed.  Ownership of a reference can be transferred.  There are three ways to
       
   856 dispose of an owned reference: pass it on, store it, or call :cfunc:`Py_DECREF`.
       
   857 Forgetting to dispose of an owned reference creates a memory leak.
       
   858 
       
   859 It is also possible to :dfn:`borrow` [#]_ a reference to an object.  The
       
   860 borrower of a reference should not call :cfunc:`Py_DECREF`.  The borrower must
       
   861 not hold on to the object longer than the owner from which it was borrowed.
       
   862 Using a borrowed reference after the owner has disposed of it risks using freed
       
   863 memory and should be avoided completely. [#]_
       
   864 
       
   865 The advantage of borrowing over owning a reference is that you don't need to
       
   866 take care of disposing of the reference on all possible paths through the code
       
   867 --- in other words, with a borrowed reference you don't run the risk of leaking
       
   868 when a premature exit is taken.  The disadvantage of borrowing over leaking is
       
   869 that there are some subtle situations where in seemingly correct code a borrowed
       
   870 reference can be used after the owner from which it was borrowed has in fact
       
   871 disposed of it.
       
   872 
       
   873 A borrowed reference can be changed into an owned reference by calling
       
   874 :cfunc:`Py_INCREF`.  This does not affect the status of the owner from which the
       
   875 reference was borrowed --- it creates a new owned reference, and gives full
       
   876 owner responsibilities (the new owner must dispose of the reference properly, as
       
   877 well as the previous owner).
       
   878 
       
   879 
       
   880 .. _ownershiprules:
       
   881 
       
   882 Ownership Rules
       
   883 ---------------
       
   884 
       
   885 Whenever an object reference is passed into or out of a function, it is part of
       
   886 the function's interface specification whether ownership is transferred with the
       
   887 reference or not.
       
   888 
       
   889 Most functions that return a reference to an object pass on ownership with the
       
   890 reference.  In particular, all functions whose function it is to create a new
       
   891 object, such as :cfunc:`PyInt_FromLong` and :cfunc:`Py_BuildValue`, pass
       
   892 ownership to the receiver.  Even if the object is not actually new, you still
       
   893 receive ownership of a new reference to that object.  For instance,
       
   894 :cfunc:`PyInt_FromLong` maintains a cache of popular values and can return a
       
   895 reference to a cached item.
       
   896 
       
   897 Many functions that extract objects from other objects also transfer ownership
       
   898 with the reference, for instance :cfunc:`PyObject_GetAttrString`.  The picture
       
   899 is less clear, here, however, since a few common routines are exceptions:
       
   900 :cfunc:`PyTuple_GetItem`, :cfunc:`PyList_GetItem`, :cfunc:`PyDict_GetItem`, and
       
   901 :cfunc:`PyDict_GetItemString` all return references that you borrow from the
       
   902 tuple, list or dictionary.
       
   903 
       
   904 The function :cfunc:`PyImport_AddModule` also returns a borrowed reference, even
       
   905 though it may actually create the object it returns: this is possible because an
       
   906 owned reference to the object is stored in ``sys.modules``.
       
   907 
       
   908 When you pass an object reference into another function, in general, the
       
   909 function borrows the reference from you --- if it needs to store it, it will use
       
   910 :cfunc:`Py_INCREF` to become an independent owner.  There are exactly two
       
   911 important exceptions to this rule: :cfunc:`PyTuple_SetItem` and
       
   912 :cfunc:`PyList_SetItem`.  These functions take over ownership of the item passed
       
   913 to them --- even if they fail!  (Note that :cfunc:`PyDict_SetItem` and friends
       
   914 don't take over ownership --- they are "normal.")
       
   915 
       
   916 When a C function is called from Python, it borrows references to its arguments
       
   917 from the caller.  The caller owns a reference to the object, so the borrowed
       
   918 reference's lifetime is guaranteed until the function returns.  Only when such a
       
   919 borrowed reference must be stored or passed on, it must be turned into an owned
       
   920 reference by calling :cfunc:`Py_INCREF`.
       
   921 
       
   922 The object reference returned from a C function that is called from Python must
       
   923 be an owned reference --- ownership is transferred from the function to its
       
   924 caller.
       
   925 
       
   926 
       
   927 .. _thinice:
       
   928 
       
   929 Thin Ice
       
   930 --------
       
   931 
       
   932 There are a few situations where seemingly harmless use of a borrowed reference
       
   933 can lead to problems.  These all have to do with implicit invocations of the
       
   934 interpreter, which can cause the owner of a reference to dispose of it.
       
   935 
       
   936 The first and most important case to know about is using :cfunc:`Py_DECREF` on
       
   937 an unrelated object while borrowing a reference to a list item.  For instance::
       
   938 
       
   939    void
       
   940    bug(PyObject *list)
       
   941    {
       
   942        PyObject *item = PyList_GetItem(list, 0);
       
   943 
       
   944        PyList_SetItem(list, 1, PyInt_FromLong(0L));
       
   945        PyObject_Print(item, stdout, 0); /* BUG! */
       
   946    }
       
   947 
       
   948 This function first borrows a reference to ``list[0]``, then replaces
       
   949 ``list[1]`` with the value ``0``, and finally prints the borrowed reference.
       
   950 Looks harmless, right?  But it's not!
       
   951 
       
   952 Let's follow the control flow into :cfunc:`PyList_SetItem`.  The list owns
       
   953 references to all its items, so when item 1 is replaced, it has to dispose of
       
   954 the original item 1.  Now let's suppose the original item 1 was an instance of a
       
   955 user-defined class, and let's further suppose that the class defined a
       
   956 :meth:`__del__` method.  If this class instance has a reference count of 1,
       
   957 disposing of it will call its :meth:`__del__` method.
       
   958 
       
   959 Since it is written in Python, the :meth:`__del__` method can execute arbitrary
       
   960 Python code.  Could it perhaps do something to invalidate the reference to
       
   961 ``item`` in :cfunc:`bug`?  You bet!  Assuming that the list passed into
       
   962 :cfunc:`bug` is accessible to the :meth:`__del__` method, it could execute a
       
   963 statement to the effect of ``del list[0]``, and assuming this was the last
       
   964 reference to that object, it would free the memory associated with it, thereby
       
   965 invalidating ``item``.
       
   966 
       
   967 The solution, once you know the source of the problem, is easy: temporarily
       
   968 increment the reference count.  The correct version of the function reads::
       
   969 
       
   970    void
       
   971    no_bug(PyObject *list)
       
   972    {
       
   973        PyObject *item = PyList_GetItem(list, 0);
       
   974 
       
   975        Py_INCREF(item);
       
   976        PyList_SetItem(list, 1, PyInt_FromLong(0L));
       
   977        PyObject_Print(item, stdout, 0);
       
   978        Py_DECREF(item);
       
   979    }
       
   980 
       
   981 This is a true story.  An older version of Python contained variants of this bug
       
   982 and someone spent a considerable amount of time in a C debugger to figure out
       
   983 why his :meth:`__del__` methods would fail...
       
   984 
       
   985 The second case of problems with a borrowed reference is a variant involving
       
   986 threads.  Normally, multiple threads in the Python interpreter can't get in each
       
   987 other's way, because there is a global lock protecting Python's entire object
       
   988 space.  However, it is possible to temporarily release this lock using the macro
       
   989 :cmacro:`Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using
       
   990 :cmacro:`Py_END_ALLOW_THREADS`.  This is common around blocking I/O calls, to
       
   991 let other threads use the processor while waiting for the I/O to complete.
       
   992 Obviously, the following function has the same problem as the previous one::
       
   993 
       
   994    void
       
   995    bug(PyObject *list)
       
   996    {
       
   997        PyObject *item = PyList_GetItem(list, 0);
       
   998        Py_BEGIN_ALLOW_THREADS
       
   999        ...some blocking I/O call...
       
  1000        Py_END_ALLOW_THREADS
       
  1001        PyObject_Print(item, stdout, 0); /* BUG! */
       
  1002    }
       
  1003 
       
  1004 
       
  1005 .. _nullpointers:
       
  1006 
       
  1007 NULL Pointers
       
  1008 -------------
       
  1009 
       
  1010 In general, functions that take object references as arguments do not expect you
       
  1011 to pass them *NULL* pointers, and will dump core (or cause later core dumps) if
       
  1012 you do so.  Functions that return object references generally return *NULL* only
       
  1013 to indicate that an exception occurred.  The reason for not testing for *NULL*
       
  1014 arguments is that functions often pass the objects they receive on to other
       
  1015 function --- if each function were to test for *NULL*, there would be a lot of
       
  1016 redundant tests and the code would run more slowly.
       
  1017 
       
  1018 It is better to test for *NULL* only at the "source:" when a pointer that may be
       
  1019 *NULL* is received, for example, from :cfunc:`malloc` or from a function that
       
  1020 may raise an exception.
       
  1021 
       
  1022 The macros :cfunc:`Py_INCREF` and :cfunc:`Py_DECREF` do not check for *NULL*
       
  1023 pointers --- however, their variants :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF`
       
  1024 do.
       
  1025 
       
  1026 The macros for checking for a particular object type (``Pytype_Check()``) don't
       
  1027 check for *NULL* pointers --- again, there is much code that calls several of
       
  1028 these in a row to test an object against various different expected types, and
       
  1029 this would generate redundant tests.  There are no variants with *NULL*
       
  1030 checking.
       
  1031 
       
  1032 The C function calling mechanism guarantees that the argument list passed to C
       
  1033 functions (``args`` in the examples) is never *NULL* --- in fact it guarantees
       
  1034 that it is always a tuple. [#]_
       
  1035 
       
  1036 It is a severe error to ever let a *NULL* pointer "escape" to the Python user.
       
  1037 
       
  1038 .. Frank Stajano:
       
  1039    A pedagogically buggy example, along the lines of the previous listing, would
       
  1040    be helpful here -- showing in more concrete terms what sort of actions could
       
  1041    cause the problem. I can't very well imagine it from the description.
       
  1042 
       
  1043 
       
  1044 .. _cplusplus:
       
  1045 
       
  1046 Writing Extensions in C++
       
  1047 =========================
       
  1048 
       
  1049 It is possible to write extension modules in C++.  Some restrictions apply.  If
       
  1050 the main program (the Python interpreter) is compiled and linked by the C
       
  1051 compiler, global or static objects with constructors cannot be used.  This is
       
  1052 not a problem if the main program is linked by the C++ compiler.  Functions that
       
  1053 will be called by the Python interpreter (in particular, module initialization
       
  1054 functions) have to be declared using ``extern "C"``. It is unnecessary to
       
  1055 enclose the Python header files in ``extern "C" {...}`` --- they use this form
       
  1056 already if the symbol ``__cplusplus`` is defined (all recent C++ compilers
       
  1057 define this symbol).
       
  1058 
       
  1059 
       
  1060 .. _using-cobjects:
       
  1061 
       
  1062 Providing a C API for an Extension Module
       
  1063 =========================================
       
  1064 
       
  1065 .. sectionauthor:: Konrad Hinsen <hinsen@cnrs-orleans.fr>
       
  1066 
       
  1067 
       
  1068 Many extension modules just provide new functions and types to be used from
       
  1069 Python, but sometimes the code in an extension module can be useful for other
       
  1070 extension modules. For example, an extension module could implement a type
       
  1071 "collection" which works like lists without order. Just like the standard Python
       
  1072 list type has a C API which permits extension modules to create and manipulate
       
  1073 lists, this new collection type should have a set of C functions for direct
       
  1074 manipulation from other extension modules.
       
  1075 
       
  1076 At first sight this seems easy: just write the functions (without declaring them
       
  1077 ``static``, of course), provide an appropriate header file, and document
       
  1078 the C API. And in fact this would work if all extension modules were always
       
  1079 linked statically with the Python interpreter. When modules are used as shared
       
  1080 libraries, however, the symbols defined in one module may not be visible to
       
  1081 another module. The details of visibility depend on the operating system; some
       
  1082 systems use one global namespace for the Python interpreter and all extension
       
  1083 modules (Windows, for example), whereas others require an explicit list of
       
  1084 imported symbols at module link time (AIX is one example), or offer a choice of
       
  1085 different strategies (most Unices). And even if symbols are globally visible,
       
  1086 the module whose functions one wishes to call might not have been loaded yet!
       
  1087 
       
  1088 Portability therefore requires not to make any assumptions about symbol
       
  1089 visibility. This means that all symbols in extension modules should be declared
       
  1090 ``static``, except for the module's initialization function, in order to
       
  1091 avoid name clashes with other extension modules (as discussed in section
       
  1092 :ref:`methodtable`). And it means that symbols that *should* be accessible from
       
  1093 other extension modules must be exported in a different way.
       
  1094 
       
  1095 Python provides a special mechanism to pass C-level information (pointers) from
       
  1096 one extension module to another one: CObjects. A CObject is a Python data type
       
  1097 which stores a pointer (:ctype:`void \*`).  CObjects can only be created and
       
  1098 accessed via their C API, but they can be passed around like any other Python
       
  1099 object. In particular,  they can be assigned to a name in an extension module's
       
  1100 namespace. Other extension modules can then import this module, retrieve the
       
  1101 value of this name, and then retrieve the pointer from the CObject.
       
  1102 
       
  1103 There are many ways in which CObjects can be used to export the C API of an
       
  1104 extension module. Each name could get its own CObject, or all C API pointers
       
  1105 could be stored in an array whose address is published in a CObject. And the
       
  1106 various tasks of storing and retrieving the pointers can be distributed in
       
  1107 different ways between the module providing the code and the client modules.
       
  1108 
       
  1109 The following example demonstrates an approach that puts most of the burden on
       
  1110 the writer of the exporting module, which is appropriate for commonly used
       
  1111 library modules. It stores all C API pointers (just one in the example!) in an
       
  1112 array of :ctype:`void` pointers which becomes the value of a CObject. The header
       
  1113 file corresponding to the module provides a macro that takes care of importing
       
  1114 the module and retrieving its C API pointers; client modules only have to call
       
  1115 this macro before accessing the C API.
       
  1116 
       
  1117 The exporting module is a modification of the :mod:`spam` module from section
       
  1118 :ref:`extending-simpleexample`. The function :func:`spam.system` does not call
       
  1119 the C library function :cfunc:`system` directly, but a function
       
  1120 :cfunc:`PySpam_System`, which would of course do something more complicated in
       
  1121 reality (such as adding "spam" to every command). This function
       
  1122 :cfunc:`PySpam_System` is also exported to other extension modules.
       
  1123 
       
  1124 The function :cfunc:`PySpam_System` is a plain C function, declared
       
  1125 ``static`` like everything else::
       
  1126 
       
  1127    static int
       
  1128    PySpam_System(const char *command)
       
  1129    {
       
  1130        return system(command);
       
  1131    }
       
  1132 
       
  1133 The function :cfunc:`spam_system` is modified in a trivial way::
       
  1134 
       
  1135    static PyObject *
       
  1136    spam_system(PyObject *self, PyObject *args)
       
  1137    {
       
  1138        const char *command;
       
  1139        int sts;
       
  1140 
       
  1141        if (!PyArg_ParseTuple(args, "s", &command))
       
  1142            return NULL;
       
  1143        sts = PySpam_System(command);
       
  1144        return Py_BuildValue("i", sts);
       
  1145    }
       
  1146 
       
  1147 In the beginning of the module, right after the line ::
       
  1148 
       
  1149    #include "Python.h"
       
  1150 
       
  1151 two more lines must be added::
       
  1152 
       
  1153    #define SPAM_MODULE
       
  1154    #include "spammodule.h"
       
  1155 
       
  1156 The ``#define`` is used to tell the header file that it is being included in the
       
  1157 exporting module, not a client module. Finally, the module's initialization
       
  1158 function must take care of initializing the C API pointer array::
       
  1159 
       
  1160    PyMODINIT_FUNC
       
  1161    initspam(void)
       
  1162    {
       
  1163        PyObject *m;
       
  1164        static void *PySpam_API[PySpam_API_pointers];
       
  1165        PyObject *c_api_object;
       
  1166 
       
  1167        m = Py_InitModule("spam", SpamMethods);
       
  1168        if (m == NULL)
       
  1169            return;
       
  1170 
       
  1171        /* Initialize the C API pointer array */
       
  1172        PySpam_API[PySpam_System_NUM] = (void *)PySpam_System;
       
  1173 
       
  1174        /* Create a CObject containing the API pointer array's address */
       
  1175        c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL);
       
  1176 
       
  1177        if (c_api_object != NULL)
       
  1178            PyModule_AddObject(m, "_C_API", c_api_object);
       
  1179    }
       
  1180 
       
  1181 Note that ``PySpam_API`` is declared ``static``; otherwise the pointer
       
  1182 array would disappear when :func:`initspam` terminates!
       
  1183 
       
  1184 The bulk of the work is in the header file :file:`spammodule.h`, which looks
       
  1185 like this::
       
  1186 
       
  1187    #ifndef Py_SPAMMODULE_H
       
  1188    #define Py_SPAMMODULE_H
       
  1189    #ifdef __cplusplus
       
  1190    extern "C" {
       
  1191    #endif
       
  1192 
       
  1193    /* Header file for spammodule */
       
  1194 
       
  1195    /* C API functions */
       
  1196    #define PySpam_System_NUM 0
       
  1197    #define PySpam_System_RETURN int
       
  1198    #define PySpam_System_PROTO (const char *command)
       
  1199 
       
  1200    /* Total number of C API pointers */
       
  1201    #define PySpam_API_pointers 1
       
  1202 
       
  1203 
       
  1204    #ifdef SPAM_MODULE
       
  1205    /* This section is used when compiling spammodule.c */
       
  1206 
       
  1207    static PySpam_System_RETURN PySpam_System PySpam_System_PROTO;
       
  1208 
       
  1209    #else
       
  1210    /* This section is used in modules that use spammodule's API */
       
  1211 
       
  1212    static void **PySpam_API;
       
  1213 
       
  1214    #define PySpam_System \
       
  1215     (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM])
       
  1216 
       
  1217    /* Return -1 and set exception on error, 0 on success. */
       
  1218    static int
       
  1219    import_spam(void)
       
  1220    {
       
  1221        PyObject *module = PyImport_ImportModule("spam");
       
  1222 
       
  1223        if (module != NULL) {
       
  1224            PyObject *c_api_object = PyObject_GetAttrString(module, "_C_API");
       
  1225            if (c_api_object == NULL)
       
  1226                return -1;
       
  1227            if (PyCObject_Check(c_api_object))
       
  1228                PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object);
       
  1229            Py_DECREF(c_api_object);
       
  1230        }
       
  1231        return 0;
       
  1232    }
       
  1233 
       
  1234    #endif
       
  1235 
       
  1236    #ifdef __cplusplus
       
  1237    }
       
  1238    #endif
       
  1239 
       
  1240    #endif /* !defined(Py_SPAMMODULE_H) */
       
  1241 
       
  1242 All that a client module must do in order to have access to the function
       
  1243 :cfunc:`PySpam_System` is to call the function (or rather macro)
       
  1244 :cfunc:`import_spam` in its initialization function::
       
  1245 
       
  1246    PyMODINIT_FUNC
       
  1247    initclient(void)
       
  1248    {
       
  1249        PyObject *m;
       
  1250 
       
  1251        m = Py_InitModule("client", ClientMethods);
       
  1252        if (m == NULL)
       
  1253            return;
       
  1254        if (import_spam() < 0)
       
  1255            return;
       
  1256        /* additional initialization can happen here */
       
  1257    }
       
  1258 
       
  1259 The main disadvantage of this approach is that the file :file:`spammodule.h` is
       
  1260 rather complicated. However, the basic structure is the same for each function
       
  1261 that is exported, so it has to be learned only once.
       
  1262 
       
  1263 Finally it should be mentioned that CObjects offer additional functionality,
       
  1264 which is especially useful for memory allocation and deallocation of the pointer
       
  1265 stored in a CObject. The details are described in the Python/C API Reference
       
  1266 Manual in the section :ref:`cobjects` and in the implementation of CObjects (files
       
  1267 :file:`Include/cobject.h` and :file:`Objects/cobject.c` in the Python source
       
  1268 code distribution).
       
  1269 
       
  1270 .. rubric:: Footnotes
       
  1271 
       
  1272 .. [#] An interface for this function already exists in the standard module :mod:`os`
       
  1273    --- it was chosen as a simple and straightforward example.
       
  1274 
       
  1275 .. [#] The metaphor of "borrowing" a reference is not completely correct: the owner
       
  1276    still has a copy of the reference.
       
  1277 
       
  1278 .. [#] Checking that the reference count is at least 1 **does not work** --- the
       
  1279    reference count itself could be in freed memory and may thus be reused for
       
  1280    another object!
       
  1281 
       
  1282 .. [#] These guarantees don't hold when you use the "old" style calling convention ---
       
  1283    this is still found in much existing code.
       
  1284