.. highlightlang:: c

.. _defining-new-types:

******************

Defining New Types

******************

.. sectionauthor:: Michael Hudson <mwh@python.net>

.. sectionauthor:: Dave Kuhlman <dkuhlman@rexx.com>

.. sectionauthor:: Jim Fulton <jim@zope.com>

As mentioned in the last chapter, Python allows the writer of an extension

module to define new types that can be manipulated from Python code, much like

strings and lists in core Python.

This is not hard; the code for all extension types follows a pattern, but there

are some details that you need to understand before you can get started.

.. note::

   The way new types are defined changed dramatically (and for the better) in

   Python 2.2.  This document documents how to define new types for Python 2.2 and

   later.  If you need to support older versions of Python, you will need to refer

   to `older versions of this documentation

   <http://www.python.org/doc/versions/>`_.

.. _dnt-basics:

The Basics

==========

The Python runtime sees all Python objects as variables of type

:ctype:`PyObject\*`.  A :ctype:`PyObject` is not a very magnificent object - it

just contains the refcount and a pointer to the object's "type object".  This is

where the action is; the type object determines which (C) functions get called

when, for instance, an attribute gets looked up on an object or it is multiplied

by another object.  These C functions are called "type methods" to distinguish

them from things like ``[].append`` (which we call "object methods").

So, if you want to define a new object type, you need to create a new type

object.

This sort of thing can only be explained by example, so here's a minimal, but

complete, module that defines a new type:

.. literalinclude:: ../includes/noddy.c

Now that's quite a bit to take in at once, but hopefully bits will seem familiar

from the last chapter.

The first bit that will be new is::

   typedef struct {

       PyObject_HEAD

   } noddy_NoddyObject;

This is what a Noddy object will contain---in this case, nothing more than every

Python object contains, namely a refcount and a pointer to a type object.  These

are the fields the ``PyObject_HEAD`` macro brings in.  The reason for the macro

is to standardize the layout and to enable special debugging fields in debug

builds.  Note that there is no semicolon after the ``PyObject_HEAD`` macro; one

is included in the macro definition.  Be wary of adding one by accident; it's

easy to do from habit, and your compiler might not complain, but someone else's

probably will!  (On Windows, MSVC is known to call this an error and refuse to

compile the code.)

For contrast, let's take a look at the corresponding definition for standard

Python integers::

   typedef struct {

       PyObject_HEAD

       long ob_ival;

   } PyIntObject;

Moving on, we come to the crunch --- the type object. ::

   static PyTypeObject noddy_NoddyType = {

       PyObject_HEAD_INIT(NULL)

       0,                         /*ob_size*/

       "noddy.Noddy",             /*tp_name*/

       sizeof(noddy_NoddyObject), /*tp_basicsize*/

       0,                         /*tp_itemsize*/

       0,                         /*tp_dealloc*/

       0,                         /*tp_print*/

       0,                         /*tp_getattr*/

       0,                         /*tp_setattr*/

       0,                         /*tp_compare*/

       0,                         /*tp_repr*/

       0,                         /*tp_as_number*/

       0,                         /*tp_as_sequence*/

       0,                         /*tp_as_mapping*/

       0,                         /*tp_hash */

       0,                         /*tp_call*/

       0,                         /*tp_str*/

       0,                         /*tp_getattro*/

       0,                         /*tp_setattro*/

       0,                         /*tp_as_buffer*/

       Py_TPFLAGS_DEFAULT,        /*tp_flags*/

       "Noddy objects",           /* tp_doc */

   };

Now if you go and look up the definition of :ctype:`PyTypeObject` in

:file:`object.h` you'll see that it has many more fields that the definition

above.  The remaining fields will be filled with zeros by the C compiler, and

it's common practice to not specify them explicitly unless you need them.

This is so important that we're going to pick the top of it apart still

further::

   PyObject_HEAD_INIT(NULL)

This line is a bit of a wart; what we'd like to write is::

   PyObject_HEAD_INIT(&PyType_Type)

as the type of a type object is "type", but this isn't strictly conforming C and

some compilers complain.  Fortunately, this member will be filled in for us by

:cfunc:`PyType_Ready`. ::

   0,                          /* ob_size */

The :attr:`ob_size` field of the header is not used; its presence in the type

structure is a historical artifact that is maintained for binary compatibility

with extension modules compiled for older versions of Python.  Always set this

field to zero. ::

   "noddy.Noddy",              /* tp_name */

The name of our type.  This will appear in the default textual representation of

our objects and in some error messages, for example::

   >>> "" + noddy.new_noddy()

   Traceback (most recent call last):

     File "<stdin>", line 1, in ?

   TypeError: cannot add type "noddy.Noddy" to string

Note that the name is a dotted name that includes both the module name and the

name of the type within the module. The module in this case is :mod:`noddy` and

the type is :class:`Noddy`, so we set the type name to :class:`noddy.Noddy`. ::

   sizeof(noddy_NoddyObject),  /* tp_basicsize */

This is so that Python knows how much memory to allocate when you call

:cfunc:`PyObject_New`.

.. note::

   If you want your type to be subclassable from Python, and your type has the same

   :attr:`tp_basicsize` as its base type, you may have problems with multiple

   inheritance.  A Python subclass of your type will have to list your type first

   in its :attr:`__bases__`, or else it will not be able to call your type's

   :meth:`__new__` method without getting an error.  You can avoid this problem by

   ensuring that your type has a larger value for :attr:`tp_basicsize` than its

   base type does.  Most of the time, this will be true anyway, because either your

   base type will be :class:`object`, or else you will be adding data members to

   your base type, and therefore increasing its size.

::

   0,                          /* tp_itemsize */

This has to do with variable length objects like lists and strings. Ignore this

for now.

Skipping a number of type methods that we don't provide, we set the class flags

to :const:`Py_TPFLAGS_DEFAULT`. ::

   Py_TPFLAGS_DEFAULT,        /*tp_flags*/

All types should include this constant in their flags.  It enables all of the

members defined by the current version of Python.

We provide a doc string for the type in :attr:`tp_doc`. ::

   "Noddy objects",           /* tp_doc */

Now we get into the type methods, the things that make your objects different

from the others.  We aren't going to implement any of these in this version of

the module.  We'll expand this example later to have more interesting behavior.

For now, all we want to be able to do is to create new :class:`Noddy` objects.

To enable object creation, we have to provide a :attr:`tp_new` implementation.

In this case, we can just use the default implementation provided by the API

function :cfunc:`PyType_GenericNew`.  We'd like to just assign this to the

:attr:`tp_new` slot, but we can't, for portability sake, On some platforms or

compilers, we can't statically initialize a structure member with a function

defined in another C module, so, instead, we'll assign the :attr:`tp_new` slot

in the module initialization function just before calling

:cfunc:`PyType_Ready`::

   noddy_NoddyType.tp_new = PyType_GenericNew;

   if (PyType_Ready(&noddy_NoddyType) < 0)

       return;

All the other type methods are *NULL*, so we'll go over them later --- that's

for a later section!

Everything else in the file should be familiar, except for some code in

:cfunc:`initnoddy`::

   if (PyType_Ready(&noddy_NoddyType) < 0)

       return;

This initializes the :class:`Noddy` type, filing in a number of members,

including :attr:`ob_type` that we initially set to *NULL*. ::

   PyModule_AddObject(m, "Noddy", (PyObject *)&noddy_NoddyType);

This adds the type to the module dictionary.  This allows us to create

:class:`Noddy` instances by calling the :class:`Noddy` class::

   >>> import noddy

   >>> mynoddy = noddy.Noddy()

That's it!  All that remains is to build it; put the above code in a file called

:file:`noddy.c` and ::

   from distutils.core import setup, Extension

   setup(name="noddy", version="1.0",

         ext_modules=[Extension("noddy", ["noddy.c"])])

in a file called :file:`setup.py`; then typing ::

   $ python setup.py build

at a shell should produce a file :file:`noddy.so` in a subdirectory; move to

that directory and fire up Python --- you should be able to ``import noddy`` and

play around with Noddy objects.

That wasn't so hard, was it?

Of course, the current Noddy type is pretty uninteresting. It has no data and

doesn't do anything. It can't even be subclassed.

Adding data and methods to the Basic example

--------------------------------------------

Let's expend the basic example to add some data and methods.  Let's also make

the type usable as a base class. We'll create a new module, :mod:`noddy2` that

adds these capabilities:

.. literalinclude:: ../includes/noddy2.c

This version of the module has a number of changes.

We've added an extra include::

   #include "structmember.h"

This include provides declarations that we use to handle attributes, as

described a bit later.

The name of the :class:`Noddy` object structure has been shortened to

:class:`Noddy`.  The type object name has been shortened to :class:`NoddyType`.

The  :class:`Noddy` type now has three data attributes, *first*, *last*, and

*number*.  The *first* and *last* variables are Python strings containing first

and last names. The *number* attribute is an integer.

The object structure is updated accordingly::

   typedef struct {

       PyObject_HEAD

       PyObject *first;

       PyObject *last;

       int number;

   } Noddy;

Because we now have data to manage, we have to be more careful about object

allocation and deallocation.  At a minimum, we need a deallocation method::

   static void

   Noddy_dealloc(Noddy* self)

   {

       Py_XDECREF(self->first);

       Py_XDECREF(self->last);

       self->ob_type->tp_free((PyObject*)self);

   }

which is assigned to the :attr:`tp_dealloc` member::

   (destructor)Noddy_dealloc, /*tp_dealloc*/

This method decrements the reference counts of the two Python attributes. We use

:cfunc:`Py_XDECREF` here because the :attr:`first` and :attr:`last` members

could be *NULL*.  It then calls the :attr:`tp_free` member of the object's type

to free the object's memory.  Note that the object's type might not be

:class:`NoddyType`, because the object may be an instance of a subclass.

We want to make sure that the first and last names are initialized to empty

strings, so we provide a new method::

   static PyObject *

   Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)

   {

       Noddy *self;

       self = (Noddy *)type->tp_alloc(type, 0);

       if (self != NULL) {

           self->first = PyString_FromString("");

           if (self->first == NULL)

             {

               Py_DECREF(self);

               return NULL;

             }

           self->last = PyString_FromString("");

           if (self->last == NULL)

             {

               Py_DECREF(self);

               return NULL;

             }

           self->number = 0;

       }

       return (PyObject *)self;

   }

and install it in the :attr:`tp_new` member::

   Noddy_new,                 /* tp_new */

The new member is responsible for creating (as opposed to initializing) objects

of the type.  It is exposed in Python as the :meth:`__new__` method.  See the

paper titled "Unifying types and classes in Python" for a detailed discussion of

the :meth:`__new__` method.  One reason to implement a new method is to assure

the initial values of instance variables.  In this case, we use the new method

to make sure that the initial values of the members :attr:`first` and

:attr:`last` are not *NULL*. If we didn't care whether the initial values were

*NULL*, we could have used :cfunc:`PyType_GenericNew` as our new method, as we

did before.  :cfunc:`PyType_GenericNew` initializes all of the instance variable

members to *NULL*.

The new method is a static method that is passed the type being instantiated and

any arguments passed when the type was called, and that returns the new object

created. New methods always accept positional and keyword arguments, but they

often ignore the arguments, leaving the argument handling to initializer

methods. Note that if the type supports subclassing, the type passed may not be

the type being defined.  The new method calls the tp_alloc slot to allocate

memory. We don't fill the :attr:`tp_alloc` slot ourselves. Rather

:cfunc:`PyType_Ready` fills it for us by inheriting it from our base class,

which is :class:`object` by default.  Most types use the default allocation.

.. note::

   If you are creating a co-operative :attr:`tp_new` (one that calls a base type's

   :attr:`tp_new` or :meth:`__new__`), you must *not* try to determine what method

   to call using method resolution order at runtime.  Always statically determine

   what type you are going to call, and call its :attr:`tp_new` directly, or via

   ``type->tp_base->tp_new``.  If you do not do this, Python subclasses of your

   type that also inherit from other Python-defined classes may not work correctly.

   (Specifically, you may not be able to create instances of such subclasses

   without getting a :exc:`TypeError`.)

We provide an initialization function::

   static int

   Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)

   {

       PyObject *first=NULL, *last=NULL, *tmp;

       static char *kwlist[] = {"first", "last", "number", NULL};

       if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,

                                         &first, &last,

                                         &self->number))

           return -1;

       if (first) {

           tmp = self->first;

           Py_INCREF(first);

           self->first = first;

           Py_XDECREF(tmp);

       }

       if (last) {

           tmp = self->last;

           Py_INCREF(last);

           self->last = last;

           Py_XDECREF(tmp);

       }

       return 0;

   }

by filling the :attr:`tp_init` slot. ::

   (initproc)Noddy_init,         /* tp_init */

The :attr:`tp_init` slot is exposed in Python as the :meth:`__init__` method. It

is used to initialize an object after it's created. Unlike the new method, we

can't guarantee that the initializer is called.  The initializer isn't called

when unpickling objects and it can be overridden.  Our initializer accepts

arguments to provide initial values for our instance. Initializers always accept

positional and keyword arguments.

Initializers can be called multiple times.  Anyone can call the :meth:`__init__`

method on our objects.  For this reason, we have to be extra careful when

assigning the new values.  We might be tempted, for example to assign the

:attr:`first` member like this::

   if (first) {

       Py_XDECREF(self->first);

       Py_INCREF(first);

       self->first = first;

   }

But this would be risky.  Our type doesn't restrict the type of the

:attr:`first` member, so it could be any kind of object.  It could have a

destructor that causes code to be executed that tries to access the

:attr:`first` member.  To be paranoid and protect ourselves against this

possibility, we almost always reassign members before decrementing their

reference counts.  When don't we have to do this?

* when we absolutely know that the reference count is greater than 1

* when we know that deallocation of the object [#]_ will not cause any calls

  back into our type's code

* when decrementing a reference count in a :attr:`tp_dealloc` handler when

  garbage-collections is not supported [#]_

We want to expose our instance variables as attributes. There are a

number of ways to do that. The simplest way is to define member definitions::

   static PyMemberDef Noddy_members[] = {

       {"first", T_OBJECT_EX, offsetof(Noddy, first), 0,

        "first name"},

       {"last", T_OBJECT_EX, offsetof(Noddy, last), 0,

        "last name"},

       {"number", T_INT, offsetof(Noddy, number), 0,

        "noddy number"},

       {NULL}  /* Sentinel */

   };

and put the definitions in the :attr:`tp_members` slot::

   Noddy_members,             /* tp_members */

Each member definition has a member name, type, offset, access flags and

documentation string. See the "Generic Attribute Management" section below for

details.

A disadvantage of this approach is that it doesn't provide a way to restrict the

types of objects that can be assigned to the Python attributes.  We expect the

first and last names to be strings, but any Python objects can be assigned.

Further, the attributes can be deleted, setting the C pointers to *NULL*.  Even

though we can make sure the members are initialized to non-*NULL* values, the

members can be set to *NULL* if the attributes are deleted.

We define a single method, :meth:`name`, that outputs the objects name as the

concatenation of the first and last names. ::

   static PyObject *

   Noddy_name(Noddy* self)

   {

       static PyObject *format = NULL;

       PyObject *args, *result;

       if (format == NULL) {

           format = PyString_FromString("%s %s");

           if (format == NULL)

               return NULL;

       }

       if (self->first == NULL) {

           PyErr_SetString(PyExc_AttributeError, "first");

           return NULL;

       }

       if (self->last == NULL) {

           PyErr_SetString(PyExc_AttributeError, "last");

           return NULL;

       }

       args = Py_BuildValue("OO", self->first, self->last);

       if (args == NULL)

           return NULL;

       result = PyString_Format(format, args);

       Py_DECREF(args);

       return result;

   }

The method is implemented as a C function that takes a :class:`Noddy` (or

:class:`Noddy` subclass) instance as the first argument.  Methods always take an

instance as the first argument. Methods often take positional and keyword

arguments as well, but in this cased we don't take any and don't need to accept

a positional argument tuple or keyword argument dictionary. This method is

equivalent to the Python method::

   def name(self):

      return "%s %s" % (self.first, self.last)

Note that we have to check for the possibility that our :attr:`first` and

:attr:`last` members are *NULL*.  This is because they can be deleted, in which

case they are set to *NULL*.  It would be better to prevent deletion of these

attributes and to restrict the attribute values to be strings.  We'll see how to

do that in the next section.

Now that we've defined the method, we need to create an array of method

definitions::

   static PyMethodDef Noddy_methods[] = {

       {"name", (PyCFunction)Noddy_name, METH_NOARGS,

        "Return the name, combining the first and last name"

       },

       {NULL}  /* Sentinel */

   };

and assign them to the :attr:`tp_methods` slot::

   Noddy_methods,             /* tp_methods */

Note that we used the :const:`METH_NOARGS` flag to indicate that the method is

passed no arguments.

Finally, we'll make our type usable as a base class.  We've written our methods

carefully so far so that they don't make any assumptions about the type of the

object being created or used, so all we need to do is to add the