.. highlightlang:: c

.. _extending-intro:

******************************

Extending Python with C or C++

******************************

It is quite easy to add new built-in modules to Python, if you know how to

program in C.  Such :dfn:`extension modules` can do two things that can't be

done directly in Python: they can implement new built-in object types, and they

can call C library functions and system calls.

To support extensions, the Python API (Application Programmers Interface)

defines a set of functions, macros and variables that provide access to most

aspects of the Python run-time system.  The Python API is incorporated in a C

source file by including the header ``"Python.h"``.

The compilation of an extension module depends on its intended use as well as on

your system setup; details are given in later chapters.

.. _extending-simpleexample:

A Simple Example

================

Let's create an extension module called ``spam`` (the favorite food of Monty

Python fans...) and let's say we want to create a Python interface to the C

library function :cfunc:`system`. [#]_ This function takes a null-terminated

character string as argument and returns an integer.  We want this function to

be callable from Python as follows::

   >>> import spam

   >>> status = spam.system("ls -l")

Begin by creating a file :file:`spammodule.c`.  (Historically, if a module is

called ``spam``, the C file containing its implementation is called

:file:`spammodule.c`; if the module name is very long, like ``spammify``, the

module name can be just :file:`spammify.c`.)

The first line of our file can be::

   #include <Python.h>

which pulls in the Python API (you can add a comment describing the purpose of

the module and a copyright notice if you like).

.. warning::

   Since Python may define some pre-processor definitions which affect the standard

   headers on some systems, you *must* include :file:`Python.h` before any standard

   headers are included.

All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or

``PY``, except those defined in standard header files. For convenience, and

since they are used extensively by the Python interpreter, ``"Python.h"``

includes a few standard header files: ``<stdio.h>``, ``<string.h>``,

``<errno.h>``, and ``<stdlib.h>``.  If the latter header file does not exist on

your system, it declares the functions :cfunc:`malloc`, :cfunc:`free` and

:cfunc:`realloc` directly.

The next thing we add to our module file is the C function that will be called

when the Python expression ``spam.system(string)`` is evaluated (we'll see

shortly how it ends up being called)::

   static PyObject *

   spam_system(PyObject *self, PyObject *args)

   {

       const char *command;

       int sts;

       if (!PyArg_ParseTuple(args, "s", &command))

           return NULL;

       sts = system(command);

       return Py_BuildValue("i", sts);

   }

There is a straightforward translation from the argument list in Python (for

example, the single expression ``"ls -l"``) to the arguments passed to the C

function.  The C function always has two arguments, conventionally named *self*

and *args*.

The *self* argument is only used when the C function implements a built-in

method, not a function. In the example, *self* will always be a *NULL* pointer,

since we are defining a function, not a method.  (This is done so that the

interpreter doesn't have to understand two different types of C functions.)

The *args* argument will be a pointer to a Python tuple object containing the

arguments.  Each item of the tuple corresponds to an argument in the call's

argument list.  The arguments are Python objects --- in order to do anything

with them in our C function we have to convert them to C values.  The function

:cfunc:`PyArg_ParseTuple` in the Python API checks the argument types and

converts them to C values.  It uses a template string to determine the required

types of the arguments as well as the types of the C variables into which to

store the converted values.  More about this later.

:cfunc:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right

type and its components have been stored in the variables whose addresses are

passed.  It returns false (zero) if an invalid argument list was passed.  In the

latter case it also raises an appropriate exception so the calling function can

return *NULL* immediately (as we saw in the example).

.. _extending-errors:

Intermezzo: Errors and Exceptions

=================================

An important convention throughout the Python interpreter is the following: when

a function fails, it should set an exception condition and return an error value

(usually a *NULL* pointer).  Exceptions are stored in a static global variable

inside the interpreter; if this variable is *NULL* no exception has occurred.  A

second global variable stores the "associated value" of the exception (the

second argument to :keyword:`raise`).  A third variable contains the stack

traceback in case the error originated in Python code.  These three variables

are the C equivalents of the Python variables ``sys.exc_type``,

``sys.exc_value`` and ``sys.exc_traceback`` (see the section on module

:mod:`sys` in the Python Library Reference).  It is important to know about them

to understand how errors are passed around.

The Python API defines a number of functions to set various types of exceptions.

The most common one is :cfunc:`PyErr_SetString`.  Its arguments are an exception

object and a C string.  The exception object is usually a predefined object like

:cdata:`PyExc_ZeroDivisionError`.  The C string indicates the cause of the error

and is converted to a Python string object and stored as the "associated value"

of the exception.

Another useful function is :cfunc:`PyErr_SetFromErrno`, which only takes an

exception argument and constructs the associated value by inspection of the

global variable :cdata:`errno`.  The most general function is

:cfunc:`PyErr_SetObject`, which takes two object arguments, the exception and

its associated value.  You don't need to :cfunc:`Py_INCREF` the objects passed

to any of these functions.

You can test non-destructively whether an exception has been set with

:cfunc:`PyErr_Occurred`.  This returns the current exception object, or *NULL*

if no exception has occurred.  You normally don't need to call

:cfunc:`PyErr_Occurred` to see whether an error occurred in a function call,

since you should be able to tell from the return value.

When a function *f* that calls another function *g* detects that the latter

fails, *f* should itself return an error value (usually *NULL* or ``-1``).  It

should *not* call one of the :cfunc:`PyErr_\*` functions --- one has already

been called by *g*. *f*'s caller is then supposed to also return an error

indication to *its* caller, again *without* calling :cfunc:`PyErr_\*`, and so on

--- the most detailed cause of the error was already reported by the function

that first detected it.  Once the error reaches the Python interpreter's main

loop, this aborts the currently executing Python code and tries to find an

exception handler specified by the Python programmer.

(There are situations where a module can actually give a more detailed error

message by calling another :cfunc:`PyErr_\*` function, and in such cases it is

fine to do so.  As a general rule, however, this is not necessary, and can cause

information about the cause of the error to be lost: most operations can fail

for a variety of reasons.)

To ignore an exception set by a function call that failed, the exception

condition must be cleared explicitly by calling :cfunc:`PyErr_Clear`.  The only

time C code should call :cfunc:`PyErr_Clear` is if it doesn't want to pass the

error on to the interpreter but wants to handle it completely by itself

(possibly by trying something else, or pretending nothing went wrong).

Every failing :cfunc:`malloc` call must be turned into an exception --- the

direct caller of :cfunc:`malloc` (or :cfunc:`realloc`) must call

:cfunc:`PyErr_NoMemory` and return a failure indicator itself.  All the

object-creating functions (for example, :cfunc:`PyInt_FromLong`) already do

this, so this note is only relevant to those who call :cfunc:`malloc` directly.

Also note that, with the important exception of :cfunc:`PyArg_ParseTuple` and

friends, functions that return an integer status usually return a positive value

or zero for success and ``-1`` for failure, like Unix system calls.

Finally, be careful to clean up garbage (by making :cfunc:`Py_XDECREF` or

:cfunc:`Py_DECREF` calls for objects you have already created) when you return

an error indicator!

The choice of which exception to raise is entirely yours.  There are predeclared

C objects corresponding to all built-in Python exceptions, such as

:cdata:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you

should choose exceptions wisely --- don't use :cdata:`PyExc_TypeError` to mean

that a file couldn't be opened (that should probably be :cdata:`PyExc_IOError`).

If something's wrong with the argument list, the :cfunc:`PyArg_ParseTuple`

function usually raises :cdata:`PyExc_TypeError`.  If you have an argument whose

value must be in a particular range or must satisfy other conditions,

:cdata:`PyExc_ValueError` is appropriate.

You can also define a new exception that is unique to your module. For this, you

usually declare a static object variable at the beginning of your file::

   static PyObject *SpamError;

and initialize it in your module's initialization function (:cfunc:`initspam`)

with an exception object (leaving out the error checking for now)::

   PyMODINIT_FUNC

   initspam(void)

   {

       PyObject *m;

       m = Py_InitModule("spam", SpamMethods);

       if (m == NULL)

           return;

       SpamError = PyErr_NewException("spam.error", NULL, NULL);

       Py_INCREF(SpamError);

       PyModule_AddObject(m, "error", SpamError);

   }

Note that the Python name for the exception object is :exc:`spam.error`.  The

:cfunc:`PyErr_NewException` function may create a class with the base class

being :exc:`Exception` (unless another class is passed in instead of *NULL*),

described in :ref:`bltin-exceptions`.

Note also that the :cdata:`SpamError` variable retains a reference to the newly

created exception class; this is intentional!  Since the exception could be

removed from the module by external code, an owned reference to the class is

needed to ensure that it will not be discarded, causing :cdata:`SpamError` to

become a dangling pointer. Should it become a dangling pointer, C code which

raises the exception could cause a core dump or other unintended side effects.

We discuss the use of PyMODINIT_FUNC as a function return type later in this

sample.

.. _backtoexample:

Back to the Example

===================

Going back to our example function, you should now be able to understand this

statement::

   if (!PyArg_ParseTuple(args, "s", &command))

       return NULL;

It returns *NULL* (the error indicator for functions returning object pointers)

if an error is detected in the argument list, relying on the exception set by

:cfunc:`PyArg_ParseTuple`.  Otherwise the string value of the argument has been

copied to the local variable :cdata:`command`.  This is a pointer assignment and

you are not supposed to modify the string to which it points (so in Standard C,

the variable :cdata:`command` should properly be declared as ``const char

*command``).

The next statement is a call to the Unix function :cfunc:`system`, passing it

the string we just got from :cfunc:`PyArg_ParseTuple`::

   sts = system(command);

Our :func:`spam.system` function must return the value of :cdata:`sts` as a

Python object.  This is done using the function :cfunc:`Py_BuildValue`, which is

something like the inverse of :cfunc:`PyArg_ParseTuple`: it takes a format

string and an arbitrary number of C values, and returns a new Python object.

More info on :cfunc:`Py_BuildValue` is given later. ::

   return Py_BuildValue("i", sts);

In this case, it will return an integer object.  (Yes, even integers are objects

on the heap in Python!)

If you have a C function that returns no useful argument (a function returning

:ctype:`void`), the corresponding Python function must return ``None``.   You

need this idiom to do so (which is implemented by the :cmacro:`Py_RETURN_NONE`

macro)::

   Py_INCREF(Py_None);

   return Py_None;

:cdata:`Py_None` is the C name for the special Python object ``None``.  It is a

genuine Python object rather than a *NULL* pointer, which means "error" in most

contexts, as we have seen.

.. _methodtable:

The Module's Method Table and Initialization Function

=====================================================

I promised to show how :cfunc:`spam_system` is called from Python programs.

First, we need to list its name and address in a "method table"::

   static PyMethodDef SpamMethods[] = {

       ...

       {"system",  spam_system, METH_VARARGS,

        "Execute a shell command."},

       ...

       {NULL, NULL, 0, NULL}        /* Sentinel */

   };

Note the third entry (``METH_VARARGS``).  This is a flag telling the interpreter

the calling convention to be used for the C function.  It should normally always

be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means

that an obsolete variant of :cfunc:`PyArg_ParseTuple` is used.

When using only ``METH_VARARGS``, the function should expect the Python-level

parameters to be passed in as a tuple acceptable for parsing via

:cfunc:`PyArg_ParseTuple`; more information on this function is provided below.

The :const:`METH_KEYWORDS` bit may be set in the third field if keyword

arguments should be passed to the function.  In this case, the C function should

accept a third ``PyObject *`` parameter which will be a dictionary of keywords.

Use :cfunc:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a

function.

The method table must be passed to the interpreter in the module's

initialization function.  The initialization function must be named

:cfunc:`initname`, where *name* is the name of the module, and should be the

only non-\ ``static`` item defined in the module file::

   PyMODINIT_FUNC

   initspam(void)

   {

       (void) Py_InitModule("spam", SpamMethods);

   }

Note that PyMODINIT_FUNC declares the function as ``void`` return type,

declares any special linkage declarations required by the platform, and for  C++

declares the function as ``extern "C"``.

When the Python program imports module :mod:`spam` for the first time,

:cfunc:`initspam` is called. (See below for comments about embedding Python.)

It calls :cfunc:`Py_InitModule`, which creates a "module object" (which is

inserted in the dictionary ``sys.modules`` under the key ``"spam"``), and

inserts built-in function objects into the newly created module based upon the

table (an array of :ctype:`PyMethodDef` structures) that was passed as its

second argument. :cfunc:`Py_InitModule` returns a pointer to the module object

that it creates (which is unused here).  It may abort with a fatal error for

certain errors, or return *NULL* if the module could not be initialized

satisfactorily.

When embedding Python, the :cfunc:`initspam` function is not called

automatically unless there's an entry in the :cdata:`_PyImport_Inittab` table.

The easiest way to handle this is to statically initialize your

statically-linked modules by directly calling :cfunc:`initspam` after the call

to :cfunc:`Py_Initialize`::

   int

   main(int argc, char *argv[])

   {

       /* Pass argv[0] to the Python interpreter */

       Py_SetProgramName(argv[0]);

       /* Initialize the Python interpreter.  Required. */

       Py_Initialize();

       /* Add a static module */

       initspam();

An example may be found in the file :file:`Demo/embed/demo.c` in the Python

source distribution.

.. note::

   Removing entries from ``sys.modules`` or importing compiled modules into

   multiple interpreters within a process (or following a :cfunc:`fork` without an

   intervening :cfunc:`exec`) can create problems for some extension modules.

   Extension module authors should exercise caution when initializing internal data

   structures. Note also that the :func:`reload` function can be used with

   extension modules, and will call the module initialization function

   (:cfunc:`initspam` in the example), but will not load the module again if it was

   loaded from a dynamically loadable object file (:file:`.so` on Unix,

   :file:`.dll` on Windows).

A more substantial example module is included in the Python source distribution

as :file:`Modules/xxmodule.c`.  This file may be used as a  template or simply

read as an example.  The :program:`modulator.py` script included in the source

distribution or Windows install provides  a simple graphical user interface for

declaring the functions and objects which a module should implement, and can

generate a template which can be filled in.  The script lives in the

:file:`Tools/modulator/` directory; see the :file:`README` file there for more

information.

.. _compilation:

Compilation and Linkage

=======================

There are two more things to do before you can use your new extension: compiling

and linking it with the Python system.  If you use dynamic loading, the details

may depend on the style of dynamic loading your system uses; see the chapters

about building extension modules (chapter :ref:`building`) and additional

information that pertains only to building on Windows (chapter

:ref:`building-on-windows`) for more information about this.

If you can't use dynamic loading, or if you want to make your module a permanent

part of the Python interpreter, you will have to change the configuration setup

and rebuild the interpreter.  Luckily, this is very simple on Unix: just place

your file (:file:`spammodule.c` for example) in the :file:`Modules/` directory

of an unpacked source distribution, add a line to the file

:file:`Modules/Setup.local` describing your file::

   spam spammodule.o

and rebuild the interpreter by running :program:`make` in the toplevel

directory.  You can also run :program:`make` in the :file:`Modules/`

subdirectory, but then you must first rebuild :file:`Makefile` there by running

':program:`make` Makefile'.  (This is necessary each time you change the

:file:`Setup` file.)

If your module requires additional libraries to link with, these can be listed

on the line in the configuration file as well, for instance::

   spam spammodule.o -lX11

.. _callingpython:

Calling Python Functions from C

===============================

So far we have concentrated on making C functions callable from Python.  The

reverse is also useful: calling Python functions from C. This is especially the

case for libraries that support so-called "callback" functions.  If a C

interface makes use of callbacks, the equivalent Python often needs to provide a

callback mechanism to the Python programmer; the implementation will require

calling the Python callback functions from a C callback.  Other uses are also

imaginable.

Fortunately, the Python interpreter is easily called recursively, and there is a

standard interface to call a Python function.  (I won't dwell on how to call the

Python parser with a particular string as input --- if you're interested, have a

look at the implementation of the :option:`-c` command line option in

:file:`Modules/main.c` from the Python source code.)

Calling a Python function is easy.  First, the Python program must somehow pass

you the Python function object.  You should provide a function (or some other

interface) to do this.  When this function is called, save a pointer to the

Python function object (be careful to :cfunc:`Py_INCREF` it!) in a global

variable --- or wherever you see fit. For example, the following function might

be part of a module definition::

   static PyObject *my_callback = NULL;

   static PyObject *

   my_set_callback(PyObject *dummy, PyObject *args)

   {

       PyObject *result = NULL;

       PyObject *temp;

       if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {

           if (!PyCallable_Check(temp)) {

               PyErr_SetString(PyExc_TypeError, "parameter must be callable");

               return NULL;

           }

           Py_XINCREF(temp);         /* Add a reference to new callback */

           Py_XDECREF(my_callback);  /* Dispose of previous callback */

           my_callback = temp;       /* Remember new callback */

           /* Boilerplate to return "None" */

           Py_INCREF(Py_None);

           result = Py_None;

       }

       return result;

   }

This function must be registered with the interpreter using the

:const:`METH_VARARGS` flag; this is described in section :ref:`methodtable`.  The

:cfunc:`PyArg_ParseTuple` function and its arguments are documented in section

:ref:`parsetuple`.

The macros :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` increment/decrement the

reference count of an object and are safe in the presence of *NULL* pointers

(but note that *temp* will not be  *NULL* in this context).  More info on them

in section :ref:`refcounts`.

.. index:: single: PyEval_CallObject()

Later, when it is time to call the function, you call the C function

:cfunc:`PyEval_CallObject`.  This function has two arguments, both pointers to

arbitrary Python objects: the Python function, and the argument list.  The

argument list must always be a tuple object, whose length is the number of

arguments.  To call the Python function with no arguments, pass in NULL, or 

an empty tuple; to call it with one argument, pass a singleton tuple.

:cfunc:`Py_BuildValue` returns a tuple when its format string consists of zero

or more format codes between parentheses.  For example::

   int arg;

   PyObject *arglist;

   PyObject *result;

   ...

   arg = 123;

   ...

   /* Time to call the callback */

   arglist = Py_BuildValue("(i)", arg);

   result = PyEval_CallObject(my_callback, arglist);

   Py_DECREF(arglist);

:cfunc:`PyEval_CallObject` returns a Python object pointer: this is the return

value of the Python function.  :cfunc:`PyEval_CallObject` is

"reference-count-neutral" with respect to its arguments.  In the example a new

tuple was created to serve as the argument list, which is :cfunc:`Py_DECREF`\

-ed immediately after the call.

The return value of :cfunc:`PyEval_CallObject` is "new": either it is a brand

new object, or it is an existing object whose reference count has been

incremented.  So, unless you want to save it in a global variable, you should

somehow :cfunc:`Py_DECREF` the result, even (especially!) if you are not

interested in its value.

Before you do this, however, it is important to check that the return value

isn't *NULL*.  If it is, the Python function terminated by raising an exception.

If the C code that called :cfunc:`PyEval_CallObject` is called from Python, it

should now return an error indication to its Python caller, so the interpreter

can print a stack trace, or the calling Python code can handle the exception.

If this is not possible or desirable, the exception should be cleared by calling

:cfunc:`PyErr_Clear`.  For example::

   if (result == NULL)

       return NULL; /* Pass error back */

   ...use result...

   Py_DECREF(result); 

Depending on the desired interface to the Python callback function, you may also

have to provide an argument list to :cfunc:`PyEval_CallObject`.  In some cases

the argument list is also provided by the Python program, through the same

interface that specified the callback function.  It can then be saved and used

in the same manner as the function object.  In other cases, you may have to

construct a new tuple to pass as the argument list.  The simplest way to do this

is to call :cfunc:`Py_BuildValue`.  For example, if you want to pass an integral

event code, you might use the following code::

   PyObject *arglist;

   ...

   arglist = Py_BuildValue("(l)", eventcode);

   result = PyEval_CallObject(my_callback, arglist);

   Py_DECREF(arglist);