|
1 .. highlightlang:: c |
|
2 |
|
3 |
|
4 .. _extending-intro: |
|
5 |
|
6 ****************************** |
|
7 Extending Python with C or C++ |
|
8 ****************************** |
|
9 |
|
10 It is quite easy to add new built-in modules to Python, if you know how to |
|
11 program in C. Such :dfn:`extension modules` can do two things that can't be |
|
12 done directly in Python: they can implement new built-in object types, and they |
|
13 can call C library functions and system calls. |
|
14 |
|
15 To support extensions, the Python API (Application Programmers Interface) |
|
16 defines a set of functions, macros and variables that provide access to most |
|
17 aspects of the Python run-time system. The Python API is incorporated in a C |
|
18 source file by including the header ``"Python.h"``. |
|
19 |
|
20 The compilation of an extension module depends on its intended use as well as on |
|
21 your system setup; details are given in later chapters. |
|
22 |
|
23 |
|
24 .. _extending-simpleexample: |
|
25 |
|
26 A Simple Example |
|
27 ================ |
|
28 |
|
29 Let's create an extension module called ``spam`` (the favorite food of Monty |
|
30 Python fans...) and let's say we want to create a Python interface to the C |
|
31 library function :cfunc:`system`. [#]_ This function takes a null-terminated |
|
32 character string as argument and returns an integer. We want this function to |
|
33 be callable from Python as follows:: |
|
34 |
|
35 >>> import spam |
|
36 >>> status = spam.system("ls -l") |
|
37 |
|
38 Begin by creating a file :file:`spammodule.c`. (Historically, if a module is |
|
39 called ``spam``, the C file containing its implementation is called |
|
40 :file:`spammodule.c`; if the module name is very long, like ``spammify``, the |
|
41 module name can be just :file:`spammify.c`.) |
|
42 |
|
43 The first line of our file can be:: |
|
44 |
|
45 #include <Python.h> |
|
46 |
|
47 which pulls in the Python API (you can add a comment describing the purpose of |
|
48 the module and a copyright notice if you like). |
|
49 |
|
50 .. warning:: |
|
51 |
|
52 Since Python may define some pre-processor definitions which affect the standard |
|
53 headers on some systems, you *must* include :file:`Python.h` before any standard |
|
54 headers are included. |
|
55 |
|
56 All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or |
|
57 ``PY``, except those defined in standard header files. For convenience, and |
|
58 since they are used extensively by the Python interpreter, ``"Python.h"`` |
|
59 includes a few standard header files: ``<stdio.h>``, ``<string.h>``, |
|
60 ``<errno.h>``, and ``<stdlib.h>``. If the latter header file does not exist on |
|
61 your system, it declares the functions :cfunc:`malloc`, :cfunc:`free` and |
|
62 :cfunc:`realloc` directly. |
|
63 |
|
64 The next thing we add to our module file is the C function that will be called |
|
65 when the Python expression ``spam.system(string)`` is evaluated (we'll see |
|
66 shortly how it ends up being called):: |
|
67 |
|
68 static PyObject * |
|
69 spam_system(PyObject *self, PyObject *args) |
|
70 { |
|
71 const char *command; |
|
72 int sts; |
|
73 |
|
74 if (!PyArg_ParseTuple(args, "s", &command)) |
|
75 return NULL; |
|
76 sts = system(command); |
|
77 return Py_BuildValue("i", sts); |
|
78 } |
|
79 |
|
80 There is a straightforward translation from the argument list in Python (for |
|
81 example, the single expression ``"ls -l"``) to the arguments passed to the C |
|
82 function. The C function always has two arguments, conventionally named *self* |
|
83 and *args*. |
|
84 |
|
85 The *self* argument is only used when the C function implements a built-in |
|
86 method, not a function. In the example, *self* will always be a *NULL* pointer, |
|
87 since we are defining a function, not a method. (This is done so that the |
|
88 interpreter doesn't have to understand two different types of C functions.) |
|
89 |
|
90 The *args* argument will be a pointer to a Python tuple object containing the |
|
91 arguments. Each item of the tuple corresponds to an argument in the call's |
|
92 argument list. The arguments are Python objects --- in order to do anything |
|
93 with them in our C function we have to convert them to C values. The function |
|
94 :cfunc:`PyArg_ParseTuple` in the Python API checks the argument types and |
|
95 converts them to C values. It uses a template string to determine the required |
|
96 types of the arguments as well as the types of the C variables into which to |
|
97 store the converted values. More about this later. |
|
98 |
|
99 :cfunc:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right |
|
100 type and its components have been stored in the variables whose addresses are |
|
101 passed. It returns false (zero) if an invalid argument list was passed. In the |
|
102 latter case it also raises an appropriate exception so the calling function can |
|
103 return *NULL* immediately (as we saw in the example). |
|
104 |
|
105 |
|
106 .. _extending-errors: |
|
107 |
|
108 Intermezzo: Errors and Exceptions |
|
109 ================================= |
|
110 |
|
111 An important convention throughout the Python interpreter is the following: when |
|
112 a function fails, it should set an exception condition and return an error value |
|
113 (usually a *NULL* pointer). Exceptions are stored in a static global variable |
|
114 inside the interpreter; if this variable is *NULL* no exception has occurred. A |
|
115 second global variable stores the "associated value" of the exception (the |
|
116 second argument to :keyword:`raise`). A third variable contains the stack |
|
117 traceback in case the error originated in Python code. These three variables |
|
118 are the C equivalents of the Python variables ``sys.exc_type``, |
|
119 ``sys.exc_value`` and ``sys.exc_traceback`` (see the section on module |
|
120 :mod:`sys` in the Python Library Reference). It is important to know about them |
|
121 to understand how errors are passed around. |
|
122 |
|
123 The Python API defines a number of functions to set various types of exceptions. |
|
124 |
|
125 The most common one is :cfunc:`PyErr_SetString`. Its arguments are an exception |
|
126 object and a C string. The exception object is usually a predefined object like |
|
127 :cdata:`PyExc_ZeroDivisionError`. The C string indicates the cause of the error |
|
128 and is converted to a Python string object and stored as the "associated value" |
|
129 of the exception. |
|
130 |
|
131 Another useful function is :cfunc:`PyErr_SetFromErrno`, which only takes an |
|
132 exception argument and constructs the associated value by inspection of the |
|
133 global variable :cdata:`errno`. The most general function is |
|
134 :cfunc:`PyErr_SetObject`, which takes two object arguments, the exception and |
|
135 its associated value. You don't need to :cfunc:`Py_INCREF` the objects passed |
|
136 to any of these functions. |
|
137 |
|
138 You can test non-destructively whether an exception has been set with |
|
139 :cfunc:`PyErr_Occurred`. This returns the current exception object, or *NULL* |
|
140 if no exception has occurred. You normally don't need to call |
|
141 :cfunc:`PyErr_Occurred` to see whether an error occurred in a function call, |
|
142 since you should be able to tell from the return value. |
|
143 |
|
144 When a function *f* that calls another function *g* detects that the latter |
|
145 fails, *f* should itself return an error value (usually *NULL* or ``-1``). It |
|
146 should *not* call one of the :cfunc:`PyErr_\*` functions --- one has already |
|
147 been called by *g*. *f*'s caller is then supposed to also return an error |
|
148 indication to *its* caller, again *without* calling :cfunc:`PyErr_\*`, and so on |
|
149 --- the most detailed cause of the error was already reported by the function |
|
150 that first detected it. Once the error reaches the Python interpreter's main |
|
151 loop, this aborts the currently executing Python code and tries to find an |
|
152 exception handler specified by the Python programmer. |
|
153 |
|
154 (There are situations where a module can actually give a more detailed error |
|
155 message by calling another :cfunc:`PyErr_\*` function, and in such cases it is |
|
156 fine to do so. As a general rule, however, this is not necessary, and can cause |
|
157 information about the cause of the error to be lost: most operations can fail |
|
158 for a variety of reasons.) |
|
159 |
|
160 To ignore an exception set by a function call that failed, the exception |
|
161 condition must be cleared explicitly by calling :cfunc:`PyErr_Clear`. The only |
|
162 time C code should call :cfunc:`PyErr_Clear` is if it doesn't want to pass the |
|
163 error on to the interpreter but wants to handle it completely by itself |
|
164 (possibly by trying something else, or pretending nothing went wrong). |
|
165 |
|
166 Every failing :cfunc:`malloc` call must be turned into an exception --- the |
|
167 direct caller of :cfunc:`malloc` (or :cfunc:`realloc`) must call |
|
168 :cfunc:`PyErr_NoMemory` and return a failure indicator itself. All the |
|
169 object-creating functions (for example, :cfunc:`PyInt_FromLong`) already do |
|
170 this, so this note is only relevant to those who call :cfunc:`malloc` directly. |
|
171 |
|
172 Also note that, with the important exception of :cfunc:`PyArg_ParseTuple` and |
|
173 friends, functions that return an integer status usually return a positive value |
|
174 or zero for success and ``-1`` for failure, like Unix system calls. |
|
175 |
|
176 Finally, be careful to clean up garbage (by making :cfunc:`Py_XDECREF` or |
|
177 :cfunc:`Py_DECREF` calls for objects you have already created) when you return |
|
178 an error indicator! |
|
179 |
|
180 The choice of which exception to raise is entirely yours. There are predeclared |
|
181 C objects corresponding to all built-in Python exceptions, such as |
|
182 :cdata:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you |
|
183 should choose exceptions wisely --- don't use :cdata:`PyExc_TypeError` to mean |
|
184 that a file couldn't be opened (that should probably be :cdata:`PyExc_IOError`). |
|
185 If something's wrong with the argument list, the :cfunc:`PyArg_ParseTuple` |
|
186 function usually raises :cdata:`PyExc_TypeError`. If you have an argument whose |
|
187 value must be in a particular range or must satisfy other conditions, |
|
188 :cdata:`PyExc_ValueError` is appropriate. |
|
189 |
|
190 You can also define a new exception that is unique to your module. For this, you |
|
191 usually declare a static object variable at the beginning of your file:: |
|
192 |
|
193 static PyObject *SpamError; |
|
194 |
|
195 and initialize it in your module's initialization function (:cfunc:`initspam`) |
|
196 with an exception object (leaving out the error checking for now):: |
|
197 |
|
198 PyMODINIT_FUNC |
|
199 initspam(void) |
|
200 { |
|
201 PyObject *m; |
|
202 |
|
203 m = Py_InitModule("spam", SpamMethods); |
|
204 if (m == NULL) |
|
205 return; |
|
206 |
|
207 SpamError = PyErr_NewException("spam.error", NULL, NULL); |
|
208 Py_INCREF(SpamError); |
|
209 PyModule_AddObject(m, "error", SpamError); |
|
210 } |
|
211 |
|
212 Note that the Python name for the exception object is :exc:`spam.error`. The |
|
213 :cfunc:`PyErr_NewException` function may create a class with the base class |
|
214 being :exc:`Exception` (unless another class is passed in instead of *NULL*), |
|
215 described in :ref:`bltin-exceptions`. |
|
216 |
|
217 Note also that the :cdata:`SpamError` variable retains a reference to the newly |
|
218 created exception class; this is intentional! Since the exception could be |
|
219 removed from the module by external code, an owned reference to the class is |
|
220 needed to ensure that it will not be discarded, causing :cdata:`SpamError` to |
|
221 become a dangling pointer. Should it become a dangling pointer, C code which |
|
222 raises the exception could cause a core dump or other unintended side effects. |
|
223 |
|
224 We discuss the use of PyMODINIT_FUNC as a function return type later in this |
|
225 sample. |
|
226 |
|
227 |
|
228 .. _backtoexample: |
|
229 |
|
230 Back to the Example |
|
231 =================== |
|
232 |
|
233 Going back to our example function, you should now be able to understand this |
|
234 statement:: |
|
235 |
|
236 if (!PyArg_ParseTuple(args, "s", &command)) |
|
237 return NULL; |
|
238 |
|
239 It returns *NULL* (the error indicator for functions returning object pointers) |
|
240 if an error is detected in the argument list, relying on the exception set by |
|
241 :cfunc:`PyArg_ParseTuple`. Otherwise the string value of the argument has been |
|
242 copied to the local variable :cdata:`command`. This is a pointer assignment and |
|
243 you are not supposed to modify the string to which it points (so in Standard C, |
|
244 the variable :cdata:`command` should properly be declared as ``const char |
|
245 *command``). |
|
246 |
|
247 The next statement is a call to the Unix function :cfunc:`system`, passing it |
|
248 the string we just got from :cfunc:`PyArg_ParseTuple`:: |
|
249 |
|
250 sts = system(command); |
|
251 |
|
252 Our :func:`spam.system` function must return the value of :cdata:`sts` as a |
|
253 Python object. This is done using the function :cfunc:`Py_BuildValue`, which is |
|
254 something like the inverse of :cfunc:`PyArg_ParseTuple`: it takes a format |
|
255 string and an arbitrary number of C values, and returns a new Python object. |
|
256 More info on :cfunc:`Py_BuildValue` is given later. :: |
|
257 |
|
258 return Py_BuildValue("i", sts); |
|
259 |
|
260 In this case, it will return an integer object. (Yes, even integers are objects |
|
261 on the heap in Python!) |
|
262 |
|
263 If you have a C function that returns no useful argument (a function returning |
|
264 :ctype:`void`), the corresponding Python function must return ``None``. You |
|
265 need this idiom to do so (which is implemented by the :cmacro:`Py_RETURN_NONE` |
|
266 macro):: |
|
267 |
|
268 Py_INCREF(Py_None); |
|
269 return Py_None; |
|
270 |
|
271 :cdata:`Py_None` is the C name for the special Python object ``None``. It is a |
|
272 genuine Python object rather than a *NULL* pointer, which means "error" in most |
|
273 contexts, as we have seen. |
|
274 |
|
275 |
|
276 .. _methodtable: |
|
277 |
|
278 The Module's Method Table and Initialization Function |
|
279 ===================================================== |
|
280 |
|
281 I promised to show how :cfunc:`spam_system` is called from Python programs. |
|
282 First, we need to list its name and address in a "method table":: |
|
283 |
|
284 static PyMethodDef SpamMethods[] = { |
|
285 ... |
|
286 {"system", spam_system, METH_VARARGS, |
|
287 "Execute a shell command."}, |
|
288 ... |
|
289 {NULL, NULL, 0, NULL} /* Sentinel */ |
|
290 }; |
|
291 |
|
292 Note the third entry (``METH_VARARGS``). This is a flag telling the interpreter |
|
293 the calling convention to be used for the C function. It should normally always |
|
294 be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means |
|
295 that an obsolete variant of :cfunc:`PyArg_ParseTuple` is used. |
|
296 |
|
297 When using only ``METH_VARARGS``, the function should expect the Python-level |
|
298 parameters to be passed in as a tuple acceptable for parsing via |
|
299 :cfunc:`PyArg_ParseTuple`; more information on this function is provided below. |
|
300 |
|
301 The :const:`METH_KEYWORDS` bit may be set in the third field if keyword |
|
302 arguments should be passed to the function. In this case, the C function should |
|
303 accept a third ``PyObject *`` parameter which will be a dictionary of keywords. |
|
304 Use :cfunc:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a |
|
305 function. |
|
306 |
|
307 The method table must be passed to the interpreter in the module's |
|
308 initialization function. The initialization function must be named |
|
309 :cfunc:`initname`, where *name* is the name of the module, and should be the |
|
310 only non-\ ``static`` item defined in the module file:: |
|
311 |
|
312 PyMODINIT_FUNC |
|
313 initspam(void) |
|
314 { |
|
315 (void) Py_InitModule("spam", SpamMethods); |
|
316 } |
|
317 |
|
318 Note that PyMODINIT_FUNC declares the function as ``void`` return type, |
|
319 declares any special linkage declarations required by the platform, and for C++ |
|
320 declares the function as ``extern "C"``. |
|
321 |
|
322 When the Python program imports module :mod:`spam` for the first time, |
|
323 :cfunc:`initspam` is called. (See below for comments about embedding Python.) |
|
324 It calls :cfunc:`Py_InitModule`, which creates a "module object" (which is |
|
325 inserted in the dictionary ``sys.modules`` under the key ``"spam"``), and |
|
326 inserts built-in function objects into the newly created module based upon the |
|
327 table (an array of :ctype:`PyMethodDef` structures) that was passed as its |
|
328 second argument. :cfunc:`Py_InitModule` returns a pointer to the module object |
|
329 that it creates (which is unused here). It may abort with a fatal error for |
|
330 certain errors, or return *NULL* if the module could not be initialized |
|
331 satisfactorily. |
|
332 |
|
333 When embedding Python, the :cfunc:`initspam` function is not called |
|
334 automatically unless there's an entry in the :cdata:`_PyImport_Inittab` table. |
|
335 The easiest way to handle this is to statically initialize your |
|
336 statically-linked modules by directly calling :cfunc:`initspam` after the call |
|
337 to :cfunc:`Py_Initialize`:: |
|
338 |
|
339 int |
|
340 main(int argc, char *argv[]) |
|
341 { |
|
342 /* Pass argv[0] to the Python interpreter */ |
|
343 Py_SetProgramName(argv[0]); |
|
344 |
|
345 /* Initialize the Python interpreter. Required. */ |
|
346 Py_Initialize(); |
|
347 |
|
348 /* Add a static module */ |
|
349 initspam(); |
|
350 |
|
351 An example may be found in the file :file:`Demo/embed/demo.c` in the Python |
|
352 source distribution. |
|
353 |
|
354 .. note:: |
|
355 |
|
356 Removing entries from ``sys.modules`` or importing compiled modules into |
|
357 multiple interpreters within a process (or following a :cfunc:`fork` without an |
|
358 intervening :cfunc:`exec`) can create problems for some extension modules. |
|
359 Extension module authors should exercise caution when initializing internal data |
|
360 structures. Note also that the :func:`reload` function can be used with |
|
361 extension modules, and will call the module initialization function |
|
362 (:cfunc:`initspam` in the example), but will not load the module again if it was |
|
363 loaded from a dynamically loadable object file (:file:`.so` on Unix, |
|
364 :file:`.dll` on Windows). |
|
365 |
|
366 A more substantial example module is included in the Python source distribution |
|
367 as :file:`Modules/xxmodule.c`. This file may be used as a template or simply |
|
368 read as an example. The :program:`modulator.py` script included in the source |
|
369 distribution or Windows install provides a simple graphical user interface for |
|
370 declaring the functions and objects which a module should implement, and can |
|
371 generate a template which can be filled in. The script lives in the |
|
372 :file:`Tools/modulator/` directory; see the :file:`README` file there for more |
|
373 information. |
|
374 |
|
375 |
|
376 .. _compilation: |
|
377 |
|
378 Compilation and Linkage |
|
379 ======================= |
|
380 |
|
381 There are two more things to do before you can use your new extension: compiling |
|
382 and linking it with the Python system. If you use dynamic loading, the details |
|
383 may depend on the style of dynamic loading your system uses; see the chapters |
|
384 about building extension modules (chapter :ref:`building`) and additional |
|
385 information that pertains only to building on Windows (chapter |
|
386 :ref:`building-on-windows`) for more information about this. |
|
387 |
|
388 If you can't use dynamic loading, or if you want to make your module a permanent |
|
389 part of the Python interpreter, you will have to change the configuration setup |
|
390 and rebuild the interpreter. Luckily, this is very simple on Unix: just place |
|
391 your file (:file:`spammodule.c` for example) in the :file:`Modules/` directory |
|
392 of an unpacked source distribution, add a line to the file |
|
393 :file:`Modules/Setup.local` describing your file:: |
|
394 |
|
395 spam spammodule.o |
|
396 |
|
397 and rebuild the interpreter by running :program:`make` in the toplevel |
|
398 directory. You can also run :program:`make` in the :file:`Modules/` |
|
399 subdirectory, but then you must first rebuild :file:`Makefile` there by running |
|
400 ':program:`make` Makefile'. (This is necessary each time you change the |
|
401 :file:`Setup` file.) |
|
402 |
|
403 If your module requires additional libraries to link with, these can be listed |
|
404 on the line in the configuration file as well, for instance:: |
|
405 |
|
406 spam spammodule.o -lX11 |
|
407 |
|
408 |
|
409 .. _callingpython: |
|
410 |
|
411 Calling Python Functions from C |
|
412 =============================== |
|
413 |
|
414 So far we have concentrated on making C functions callable from Python. The |
|
415 reverse is also useful: calling Python functions from C. This is especially the |
|
416 case for libraries that support so-called "callback" functions. If a C |
|
417 interface makes use of callbacks, the equivalent Python often needs to provide a |
|
418 callback mechanism to the Python programmer; the implementation will require |
|
419 calling the Python callback functions from a C callback. Other uses are also |
|
420 imaginable. |
|
421 |
|
422 Fortunately, the Python interpreter is easily called recursively, and there is a |
|
423 standard interface to call a Python function. (I won't dwell on how to call the |
|
424 Python parser with a particular string as input --- if you're interested, have a |
|
425 look at the implementation of the :option:`-c` command line option in |
|
426 :file:`Modules/main.c` from the Python source code.) |
|
427 |
|
428 Calling a Python function is easy. First, the Python program must somehow pass |
|
429 you the Python function object. You should provide a function (or some other |
|
430 interface) to do this. When this function is called, save a pointer to the |
|
431 Python function object (be careful to :cfunc:`Py_INCREF` it!) in a global |
|
432 variable --- or wherever you see fit. For example, the following function might |
|
433 be part of a module definition:: |
|
434 |
|
435 static PyObject *my_callback = NULL; |
|
436 |
|
437 static PyObject * |
|
438 my_set_callback(PyObject *dummy, PyObject *args) |
|
439 { |
|
440 PyObject *result = NULL; |
|
441 PyObject *temp; |
|
442 |
|
443 if (PyArg_ParseTuple(args, "O:set_callback", &temp)) { |
|
444 if (!PyCallable_Check(temp)) { |
|
445 PyErr_SetString(PyExc_TypeError, "parameter must be callable"); |
|
446 return NULL; |
|
447 } |
|
448 Py_XINCREF(temp); /* Add a reference to new callback */ |
|
449 Py_XDECREF(my_callback); /* Dispose of previous callback */ |
|
450 my_callback = temp; /* Remember new callback */ |
|
451 /* Boilerplate to return "None" */ |
|
452 Py_INCREF(Py_None); |
|
453 result = Py_None; |
|
454 } |
|
455 return result; |
|
456 } |
|
457 |
|
458 This function must be registered with the interpreter using the |
|
459 :const:`METH_VARARGS` flag; this is described in section :ref:`methodtable`. The |
|
460 :cfunc:`PyArg_ParseTuple` function and its arguments are documented in section |
|
461 :ref:`parsetuple`. |
|
462 |
|
463 The macros :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` increment/decrement the |
|
464 reference count of an object and are safe in the presence of *NULL* pointers |
|
465 (but note that *temp* will not be *NULL* in this context). More info on them |
|
466 in section :ref:`refcounts`. |
|
467 |
|
468 .. index:: single: PyEval_CallObject() |
|
469 |
|
470 Later, when it is time to call the function, you call the C function |
|
471 :cfunc:`PyEval_CallObject`. This function has two arguments, both pointers to |
|
472 arbitrary Python objects: the Python function, and the argument list. The |
|
473 argument list must always be a tuple object, whose length is the number of |
|
474 arguments. To call the Python function with no arguments, pass in NULL, or |
|
475 an empty tuple; to call it with one argument, pass a singleton tuple. |
|
476 :cfunc:`Py_BuildValue` returns a tuple when its format string consists of zero |
|
477 or more format codes between parentheses. For example:: |
|
478 |
|
479 int arg; |
|
480 PyObject *arglist; |
|
481 PyObject *result; |
|
482 ... |
|
483 arg = 123; |
|
484 ... |
|
485 /* Time to call the callback */ |
|
486 arglist = Py_BuildValue("(i)", arg); |
|
487 result = PyEval_CallObject(my_callback, arglist); |
|
488 Py_DECREF(arglist); |
|
489 |
|
490 :cfunc:`PyEval_CallObject` returns a Python object pointer: this is the return |
|
491 value of the Python function. :cfunc:`PyEval_CallObject` is |
|
492 "reference-count-neutral" with respect to its arguments. In the example a new |
|
493 tuple was created to serve as the argument list, which is :cfunc:`Py_DECREF`\ |
|
494 -ed immediately after the call. |
|
495 |
|
496 The return value of :cfunc:`PyEval_CallObject` is "new": either it is a brand |
|
497 new object, or it is an existing object whose reference count has been |
|
498 incremented. So, unless you want to save it in a global variable, you should |
|
499 somehow :cfunc:`Py_DECREF` the result, even (especially!) if you are not |
|
500 interested in its value. |
|
501 |
|
502 Before you do this, however, it is important to check that the return value |
|
503 isn't *NULL*. If it is, the Python function terminated by raising an exception. |
|
504 If the C code that called :cfunc:`PyEval_CallObject` is called from Python, it |
|
505 should now return an error indication to its Python caller, so the interpreter |
|
506 can print a stack trace, or the calling Python code can handle the exception. |
|
507 If this is not possible or desirable, the exception should be cleared by calling |
|
508 :cfunc:`PyErr_Clear`. For example:: |
|
509 |
|
510 if (result == NULL) |
|
511 return NULL; /* Pass error back */ |
|
512 ...use result... |
|
513 Py_DECREF(result); |
|
514 |
|
515 Depending on the desired interface to the Python callback function, you may also |
|
516 have to provide an argument list to :cfunc:`PyEval_CallObject`. In some cases |
|
517 the argument list is also provided by the Python program, through the same |
|
518 interface that specified the callback function. It can then be saved and used |
|
519 in the same manner as the function object. In other cases, you may have to |
|
520 construct a new tuple to pass as the argument list. The simplest way to do this |
|
521 is to call :cfunc:`Py_BuildValue`. For example, if you want to pass an integral |
|
522 event code, you might use the following code:: |
|
523 |
|
524 PyObject *arglist; |
|
525 ... |
|
526 arglist = Py_BuildValue("(l)", eventcode); |
|
527 result = PyEval_CallObject(my_callback, arglist); |
|
528 Py_DECREF(arglist); |
|
529 if (result == NULL) |
|
530 return NULL; /* Pass error back */ |
|
531 /* Here maybe use the result */ |
|
532 Py_DECREF(result); |
|
533 |
|
534 Note the placement of ``Py_DECREF(arglist)`` immediately after the call, before |
|
535 the error check! Also note that strictly speaking this code is not complete: |
|
536 :cfunc:`Py_BuildValue` may run out of memory, and this should be checked. |
|
537 |
|
538 You may also call a function with keyword arguments by using |
|
539 :cfunc:`PyEval_CallObjectWithKeywords`. As in the above example, we use |
|
540 :cfunc:`Py_BuildValue` to construct the dictionary. :: |
|
541 |
|
542 PyObject *dict; |
|
543 ... |
|
544 dict = Py_BuildValue("{s:i}", "name", val); |
|
545 result = PyEval_CallObjectWithKeywords(my_callback, NULL, dict); |
|
546 Py_DECREF(dict); |
|
547 if (result == NULL) |
|
548 return NULL; /* Pass error back */ |
|
549 /* Here maybe use the result */ |
|
550 Py_DECREF(result); |
|
551 |
|
552 .. _parsetuple: |
|
553 |
|
554 Extracting Parameters in Extension Functions |
|
555 ============================================ |
|
556 |
|
557 .. index:: single: PyArg_ParseTuple() |
|
558 |
|
559 The :cfunc:`PyArg_ParseTuple` function is declared as follows:: |
|
560 |
|
561 int PyArg_ParseTuple(PyObject *arg, char *format, ...); |
|
562 |
|
563 The *arg* argument must be a tuple object containing an argument list passed |
|
564 from Python to a C function. The *format* argument must be a format string, |
|
565 whose syntax is explained in :ref:`arg-parsing` in the Python/C API Reference |
|
566 Manual. The remaining arguments must be addresses of variables whose type is |
|
567 determined by the format string. |
|
568 |
|
569 Note that while :cfunc:`PyArg_ParseTuple` checks that the Python arguments have |
|
570 the required types, it cannot check the validity of the addresses of C variables |
|
571 passed to the call: if you make mistakes there, your code will probably crash or |
|
572 at least overwrite random bits in memory. So be careful! |
|
573 |
|
574 Note that any Python object references which are provided to the caller are |
|
575 *borrowed* references; do not decrement their reference count! |
|
576 |
|
577 Some example calls:: |
|
578 |
|
579 int ok; |
|
580 int i, j; |
|
581 long k, l; |
|
582 const char *s; |
|
583 int size; |
|
584 |
|
585 ok = PyArg_ParseTuple(args, ""); /* No arguments */ |
|
586 /* Python call: f() */ |
|
587 |
|
588 :: |
|
589 |
|
590 ok = PyArg_ParseTuple(args, "s", &s); /* A string */ |
|
591 /* Possible Python call: f('whoops!') */ |
|
592 |
|
593 :: |
|
594 |
|
595 ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */ |
|
596 /* Possible Python call: f(1, 2, 'three') */ |
|
597 |
|
598 :: |
|
599 |
|
600 ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size); |
|
601 /* A pair of ints and a string, whose size is also returned */ |
|
602 /* Possible Python call: f((1, 2), 'three') */ |
|
603 |
|
604 :: |
|
605 |
|
606 { |
|
607 const char *file; |
|
608 const char *mode = "r"; |
|
609 int bufsize = 0; |
|
610 ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize); |
|
611 /* A string, and optionally another string and an integer */ |
|
612 /* Possible Python calls: |
|
613 f('spam') |
|
614 f('spam', 'w') |
|
615 f('spam', 'wb', 100000) */ |
|
616 } |
|
617 |
|
618 :: |
|
619 |
|
620 { |
|
621 int left, top, right, bottom, h, v; |
|
622 ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)", |
|
623 &left, &top, &right, &bottom, &h, &v); |
|
624 /* A rectangle and a point */ |
|
625 /* Possible Python call: |
|
626 f(((0, 0), (400, 300)), (10, 10)) */ |
|
627 } |
|
628 |
|
629 :: |
|
630 |
|
631 { |
|
632 Py_complex c; |
|
633 ok = PyArg_ParseTuple(args, "D:myfunction", &c); |
|
634 /* a complex, also providing a function name for errors */ |
|
635 /* Possible Python call: myfunction(1+2j) */ |
|
636 } |
|
637 |
|
638 |
|
639 .. _parsetupleandkeywords: |
|
640 |
|
641 Keyword Parameters for Extension Functions |
|
642 ========================================== |
|
643 |
|
644 .. index:: single: PyArg_ParseTupleAndKeywords() |
|
645 |
|
646 The :cfunc:`PyArg_ParseTupleAndKeywords` function is declared as follows:: |
|
647 |
|
648 int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict, |
|
649 char *format, char *kwlist[], ...); |
|
650 |
|
651 The *arg* and *format* parameters are identical to those of the |
|
652 :cfunc:`PyArg_ParseTuple` function. The *kwdict* parameter is the dictionary of |
|
653 keywords received as the third parameter from the Python runtime. The *kwlist* |
|
654 parameter is a *NULL*-terminated list of strings which identify the parameters; |
|
655 the names are matched with the type information from *format* from left to |
|
656 right. On success, :cfunc:`PyArg_ParseTupleAndKeywords` returns true, otherwise |
|
657 it returns false and raises an appropriate exception. |
|
658 |
|
659 .. note:: |
|
660 |
|
661 Nested tuples cannot be parsed when using keyword arguments! Keyword parameters |
|
662 passed in which are not present in the *kwlist* will cause :exc:`TypeError` to |
|
663 be raised. |
|
664 |
|
665 .. index:: single: Philbrick, Geoff |
|
666 |
|
667 Here is an example module which uses keywords, based on an example by Geoff |
|
668 Philbrick (philbrick@hks.com):: |
|
669 |
|
670 #include "Python.h" |
|
671 |
|
672 static PyObject * |
|
673 keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds) |
|
674 { |
|
675 int voltage; |
|
676 char *state = "a stiff"; |
|
677 char *action = "voom"; |
|
678 char *type = "Norwegian Blue"; |
|
679 |
|
680 static char *kwlist[] = {"voltage", "state", "action", "type", NULL}; |
|
681 |
|
682 if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist, |
|
683 &voltage, &state, &action, &type)) |
|
684 return NULL; |
|
685 |
|
686 printf("-- This parrot wouldn't %s if you put %i Volts through it.\n", |
|
687 action, voltage); |
|
688 printf("-- Lovely plumage, the %s -- It's %s!\n", type, state); |
|
689 |
|
690 Py_INCREF(Py_None); |
|
691 |
|
692 return Py_None; |
|
693 } |
|
694 |
|
695 static PyMethodDef keywdarg_methods[] = { |
|
696 /* The cast of the function is necessary since PyCFunction values |
|
697 * only take two PyObject* parameters, and keywdarg_parrot() takes |
|
698 * three. |
|
699 */ |
|
700 {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS, |
|
701 "Print a lovely skit to standard output."}, |
|
702 {NULL, NULL, 0, NULL} /* sentinel */ |
|
703 }; |
|
704 |
|
705 :: |
|
706 |
|
707 void |
|
708 initkeywdarg(void) |
|
709 { |
|
710 /* Create the module and add the functions */ |
|
711 Py_InitModule("keywdarg", keywdarg_methods); |
|
712 } |
|
713 |
|
714 |
|
715 .. _buildvalue: |
|
716 |
|
717 Building Arbitrary Values |
|
718 ========================= |
|
719 |
|
720 This function is the counterpart to :cfunc:`PyArg_ParseTuple`. It is declared |
|
721 as follows:: |
|
722 |
|
723 PyObject *Py_BuildValue(char *format, ...); |
|
724 |
|
725 It recognizes a set of format units similar to the ones recognized by |
|
726 :cfunc:`PyArg_ParseTuple`, but the arguments (which are input to the function, |
|
727 not output) must not be pointers, just values. It returns a new Python object, |
|
728 suitable for returning from a C function called from Python. |
|
729 |
|
730 One difference with :cfunc:`PyArg_ParseTuple`: while the latter requires its |
|
731 first argument to be a tuple (since Python argument lists are always represented |
|
732 as tuples internally), :cfunc:`Py_BuildValue` does not always build a tuple. It |
|
733 builds a tuple only if its format string contains two or more format units. If |
|
734 the format string is empty, it returns ``None``; if it contains exactly one |
|
735 format unit, it returns whatever object is described by that format unit. To |
|
736 force it to return a tuple of size 0 or one, parenthesize the format string. |
|
737 |
|
738 Examples (to the left the call, to the right the resulting Python value):: |
|
739 |
|
740 Py_BuildValue("") None |
|
741 Py_BuildValue("i", 123) 123 |
|
742 Py_BuildValue("iii", 123, 456, 789) (123, 456, 789) |
|
743 Py_BuildValue("s", "hello") 'hello' |
|
744 Py_BuildValue("ss", "hello", "world") ('hello', 'world') |
|
745 Py_BuildValue("s#", "hello", 4) 'hell' |
|
746 Py_BuildValue("()") () |
|
747 Py_BuildValue("(i)", 123) (123,) |
|
748 Py_BuildValue("(ii)", 123, 456) (123, 456) |
|
749 Py_BuildValue("(i,i)", 123, 456) (123, 456) |
|
750 Py_BuildValue("[i,i]", 123, 456) [123, 456] |
|
751 Py_BuildValue("{s:i,s:i}", |
|
752 "abc", 123, "def", 456) {'abc': 123, 'def': 456} |
|
753 Py_BuildValue("((ii)(ii)) (ii)", |
|
754 1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6)) |
|
755 |
|
756 |
|
757 .. _refcounts: |
|
758 |
|
759 Reference Counts |
|
760 ================ |
|
761 |
|
762 In languages like C or C++, the programmer is responsible for dynamic allocation |
|
763 and deallocation of memory on the heap. In C, this is done using the functions |
|
764 :cfunc:`malloc` and :cfunc:`free`. In C++, the operators ``new`` and |
|
765 ``delete`` are used with essentially the same meaning and we'll restrict |
|
766 the following discussion to the C case. |
|
767 |
|
768 Every block of memory allocated with :cfunc:`malloc` should eventually be |
|
769 returned to the pool of available memory by exactly one call to :cfunc:`free`. |
|
770 It is important to call :cfunc:`free` at the right time. If a block's address |
|
771 is forgotten but :cfunc:`free` is not called for it, the memory it occupies |
|
772 cannot be reused until the program terminates. This is called a :dfn:`memory |
|
773 leak`. On the other hand, if a program calls :cfunc:`free` for a block and then |
|
774 continues to use the block, it creates a conflict with re-use of the block |
|
775 through another :cfunc:`malloc` call. This is called :dfn:`using freed memory`. |
|
776 It has the same bad consequences as referencing uninitialized data --- core |
|
777 dumps, wrong results, mysterious crashes. |
|
778 |
|
779 Common causes of memory leaks are unusual paths through the code. For instance, |
|
780 a function may allocate a block of memory, do some calculation, and then free |
|
781 the block again. Now a change in the requirements for the function may add a |
|
782 test to the calculation that detects an error condition and can return |
|
783 prematurely from the function. It's easy to forget to free the allocated memory |
|
784 block when taking this premature exit, especially when it is added later to the |
|
785 code. Such leaks, once introduced, often go undetected for a long time: the |
|
786 error exit is taken only in a small fraction of all calls, and most modern |
|
787 machines have plenty of virtual memory, so the leak only becomes apparent in a |
|
788 long-running process that uses the leaking function frequently. Therefore, it's |
|
789 important to prevent leaks from happening by having a coding convention or |
|
790 strategy that minimizes this kind of errors. |
|
791 |
|
792 Since Python makes heavy use of :cfunc:`malloc` and :cfunc:`free`, it needs a |
|
793 strategy to avoid memory leaks as well as the use of freed memory. The chosen |
|
794 method is called :dfn:`reference counting`. The principle is simple: every |
|
795 object contains a counter, which is incremented when a reference to the object |
|
796 is stored somewhere, and which is decremented when a reference to it is deleted. |
|
797 When the counter reaches zero, the last reference to the object has been deleted |
|
798 and the object is freed. |
|
799 |
|
800 An alternative strategy is called :dfn:`automatic garbage collection`. |
|
801 (Sometimes, reference counting is also referred to as a garbage collection |
|
802 strategy, hence my use of "automatic" to distinguish the two.) The big |
|
803 advantage of automatic garbage collection is that the user doesn't need to call |
|
804 :cfunc:`free` explicitly. (Another claimed advantage is an improvement in speed |
|
805 or memory usage --- this is no hard fact however.) The disadvantage is that for |
|
806 C, there is no truly portable automatic garbage collector, while reference |
|
807 counting can be implemented portably (as long as the functions :cfunc:`malloc` |
|
808 and :cfunc:`free` are available --- which the C Standard guarantees). Maybe some |
|
809 day a sufficiently portable automatic garbage collector will be available for C. |
|
810 Until then, we'll have to live with reference counts. |
|
811 |
|
812 While Python uses the traditional reference counting implementation, it also |
|
813 offers a cycle detector that works to detect reference cycles. This allows |
|
814 applications to not worry about creating direct or indirect circular references; |
|
815 these are the weakness of garbage collection implemented using only reference |
|
816 counting. Reference cycles consist of objects which contain (possibly indirect) |
|
817 references to themselves, so that each object in the cycle has a reference count |
|
818 which is non-zero. Typical reference counting implementations are not able to |
|
819 reclaim the memory belonging to any objects in a reference cycle, or referenced |
|
820 from the objects in the cycle, even though there are no further references to |
|
821 the cycle itself. |
|
822 |
|
823 The cycle detector is able to detect garbage cycles and can reclaim them so long |
|
824 as there are no finalizers implemented in Python (:meth:`__del__` methods). |
|
825 When there are such finalizers, the detector exposes the cycles through the |
|
826 :mod:`gc` module (specifically, the |
|
827 ``garbage`` variable in that module). The :mod:`gc` module also exposes a way |
|
828 to run the detector (the :func:`collect` function), as well as configuration |
|
829 interfaces and the ability to disable the detector at runtime. The cycle |
|
830 detector is considered an optional component; though it is included by default, |
|
831 it can be disabled at build time using the :option:`--without-cycle-gc` option |
|
832 to the :program:`configure` script on Unix platforms (including Mac OS X) or by |
|
833 removing the definition of ``WITH_CYCLE_GC`` in the :file:`pyconfig.h` header on |
|
834 other platforms. If the cycle detector is disabled in this way, the :mod:`gc` |
|
835 module will not be available. |
|
836 |
|
837 |
|
838 .. _refcountsinpython: |
|
839 |
|
840 Reference Counting in Python |
|
841 ---------------------------- |
|
842 |
|
843 There are two macros, ``Py_INCREF(x)`` and ``Py_DECREF(x)``, which handle the |
|
844 incrementing and decrementing of the reference count. :cfunc:`Py_DECREF` also |
|
845 frees the object when the count reaches zero. For flexibility, it doesn't call |
|
846 :cfunc:`free` directly --- rather, it makes a call through a function pointer in |
|
847 the object's :dfn:`type object`. For this purpose (and others), every object |
|
848 also contains a pointer to its type object. |
|
849 |
|
850 The big question now remains: when to use ``Py_INCREF(x)`` and ``Py_DECREF(x)``? |
|
851 Let's first introduce some terms. Nobody "owns" an object; however, you can |
|
852 :dfn:`own a reference` to an object. An object's reference count is now defined |
|
853 as the number of owned references to it. The owner of a reference is |
|
854 responsible for calling :cfunc:`Py_DECREF` when the reference is no longer |
|
855 needed. Ownership of a reference can be transferred. There are three ways to |
|
856 dispose of an owned reference: pass it on, store it, or call :cfunc:`Py_DECREF`. |
|
857 Forgetting to dispose of an owned reference creates a memory leak. |
|
858 |
|
859 It is also possible to :dfn:`borrow` [#]_ a reference to an object. The |
|
860 borrower of a reference should not call :cfunc:`Py_DECREF`. The borrower must |
|
861 not hold on to the object longer than the owner from which it was borrowed. |
|
862 Using a borrowed reference after the owner has disposed of it risks using freed |
|
863 memory and should be avoided completely. [#]_ |
|
864 |
|
865 The advantage of borrowing over owning a reference is that you don't need to |
|
866 take care of disposing of the reference on all possible paths through the code |
|
867 --- in other words, with a borrowed reference you don't run the risk of leaking |
|
868 when a premature exit is taken. The disadvantage of borrowing over leaking is |
|
869 that there are some subtle situations where in seemingly correct code a borrowed |
|
870 reference can be used after the owner from which it was borrowed has in fact |
|
871 disposed of it. |
|
872 |
|
873 A borrowed reference can be changed into an owned reference by calling |
|
874 :cfunc:`Py_INCREF`. This does not affect the status of the owner from which the |
|
875 reference was borrowed --- it creates a new owned reference, and gives full |
|
876 owner responsibilities (the new owner must dispose of the reference properly, as |
|
877 well as the previous owner). |
|
878 |
|
879 |
|
880 .. _ownershiprules: |
|
881 |
|
882 Ownership Rules |
|
883 --------------- |
|
884 |
|
885 Whenever an object reference is passed into or out of a function, it is part of |
|
886 the function's interface specification whether ownership is transferred with the |
|
887 reference or not. |
|
888 |
|
889 Most functions that return a reference to an object pass on ownership with the |
|
890 reference. In particular, all functions whose function it is to create a new |
|
891 object, such as :cfunc:`PyInt_FromLong` and :cfunc:`Py_BuildValue`, pass |
|
892 ownership to the receiver. Even if the object is not actually new, you still |
|
893 receive ownership of a new reference to that object. For instance, |
|
894 :cfunc:`PyInt_FromLong` maintains a cache of popular values and can return a |
|
895 reference to a cached item. |
|
896 |
|
897 Many functions that extract objects from other objects also transfer ownership |
|
898 with the reference, for instance :cfunc:`PyObject_GetAttrString`. The picture |
|
899 is less clear, here, however, since a few common routines are exceptions: |
|
900 :cfunc:`PyTuple_GetItem`, :cfunc:`PyList_GetItem`, :cfunc:`PyDict_GetItem`, and |
|
901 :cfunc:`PyDict_GetItemString` all return references that you borrow from the |
|
902 tuple, list or dictionary. |
|
903 |
|
904 The function :cfunc:`PyImport_AddModule` also returns a borrowed reference, even |
|
905 though it may actually create the object it returns: this is possible because an |
|
906 owned reference to the object is stored in ``sys.modules``. |
|
907 |
|
908 When you pass an object reference into another function, in general, the |
|
909 function borrows the reference from you --- if it needs to store it, it will use |
|
910 :cfunc:`Py_INCREF` to become an independent owner. There are exactly two |
|
911 important exceptions to this rule: :cfunc:`PyTuple_SetItem` and |
|
912 :cfunc:`PyList_SetItem`. These functions take over ownership of the item passed |
|
913 to them --- even if they fail! (Note that :cfunc:`PyDict_SetItem` and friends |
|
914 don't take over ownership --- they are "normal.") |
|
915 |
|
916 When a C function is called from Python, it borrows references to its arguments |
|
917 from the caller. The caller owns a reference to the object, so the borrowed |
|
918 reference's lifetime is guaranteed until the function returns. Only when such a |
|
919 borrowed reference must be stored or passed on, it must be turned into an owned |
|
920 reference by calling :cfunc:`Py_INCREF`. |
|
921 |
|
922 The object reference returned from a C function that is called from Python must |
|
923 be an owned reference --- ownership is transferred from the function to its |
|
924 caller. |
|
925 |
|
926 |
|
927 .. _thinice: |
|
928 |
|
929 Thin Ice |
|
930 -------- |
|
931 |
|
932 There are a few situations where seemingly harmless use of a borrowed reference |
|
933 can lead to problems. These all have to do with implicit invocations of the |
|
934 interpreter, which can cause the owner of a reference to dispose of it. |
|
935 |
|
936 The first and most important case to know about is using :cfunc:`Py_DECREF` on |
|
937 an unrelated object while borrowing a reference to a list item. For instance:: |
|
938 |
|
939 void |
|
940 bug(PyObject *list) |
|
941 { |
|
942 PyObject *item = PyList_GetItem(list, 0); |
|
943 |
|
944 PyList_SetItem(list, 1, PyInt_FromLong(0L)); |
|
945 PyObject_Print(item, stdout, 0); /* BUG! */ |
|
946 } |
|
947 |
|
948 This function first borrows a reference to ``list[0]``, then replaces |
|
949 ``list[1]`` with the value ``0``, and finally prints the borrowed reference. |
|
950 Looks harmless, right? But it's not! |
|
951 |
|
952 Let's follow the control flow into :cfunc:`PyList_SetItem`. The list owns |
|
953 references to all its items, so when item 1 is replaced, it has to dispose of |
|
954 the original item 1. Now let's suppose the original item 1 was an instance of a |
|
955 user-defined class, and let's further suppose that the class defined a |
|
956 :meth:`__del__` method. If this class instance has a reference count of 1, |
|
957 disposing of it will call its :meth:`__del__` method. |
|
958 |
|
959 Since it is written in Python, the :meth:`__del__` method can execute arbitrary |
|
960 Python code. Could it perhaps do something to invalidate the reference to |
|
961 ``item`` in :cfunc:`bug`? You bet! Assuming that the list passed into |
|
962 :cfunc:`bug` is accessible to the :meth:`__del__` method, it could execute a |
|
963 statement to the effect of ``del list[0]``, and assuming this was the last |
|
964 reference to that object, it would free the memory associated with it, thereby |
|
965 invalidating ``item``. |
|
966 |
|
967 The solution, once you know the source of the problem, is easy: temporarily |
|
968 increment the reference count. The correct version of the function reads:: |
|
969 |
|
970 void |
|
971 no_bug(PyObject *list) |
|
972 { |
|
973 PyObject *item = PyList_GetItem(list, 0); |
|
974 |
|
975 Py_INCREF(item); |
|
976 PyList_SetItem(list, 1, PyInt_FromLong(0L)); |
|
977 PyObject_Print(item, stdout, 0); |
|
978 Py_DECREF(item); |
|
979 } |
|
980 |
|
981 This is a true story. An older version of Python contained variants of this bug |
|
982 and someone spent a considerable amount of time in a C debugger to figure out |
|
983 why his :meth:`__del__` methods would fail... |
|
984 |
|
985 The second case of problems with a borrowed reference is a variant involving |
|
986 threads. Normally, multiple threads in the Python interpreter can't get in each |
|
987 other's way, because there is a global lock protecting Python's entire object |
|
988 space. However, it is possible to temporarily release this lock using the macro |
|
989 :cmacro:`Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using |
|
990 :cmacro:`Py_END_ALLOW_THREADS`. This is common around blocking I/O calls, to |
|
991 let other threads use the processor while waiting for the I/O to complete. |
|
992 Obviously, the following function has the same problem as the previous one:: |
|
993 |
|
994 void |
|
995 bug(PyObject *list) |
|
996 { |
|
997 PyObject *item = PyList_GetItem(list, 0); |
|
998 Py_BEGIN_ALLOW_THREADS |
|
999 ...some blocking I/O call... |
|
1000 Py_END_ALLOW_THREADS |
|
1001 PyObject_Print(item, stdout, 0); /* BUG! */ |
|
1002 } |
|
1003 |
|
1004 |
|
1005 .. _nullpointers: |
|
1006 |
|
1007 NULL Pointers |
|
1008 ------------- |
|
1009 |
|
1010 In general, functions that take object references as arguments do not expect you |
|
1011 to pass them *NULL* pointers, and will dump core (or cause later core dumps) if |
|
1012 you do so. Functions that return object references generally return *NULL* only |
|
1013 to indicate that an exception occurred. The reason for not testing for *NULL* |
|
1014 arguments is that functions often pass the objects they receive on to other |
|
1015 function --- if each function were to test for *NULL*, there would be a lot of |
|
1016 redundant tests and the code would run more slowly. |
|
1017 |
|
1018 It is better to test for *NULL* only at the "source:" when a pointer that may be |
|
1019 *NULL* is received, for example, from :cfunc:`malloc` or from a function that |
|
1020 may raise an exception. |
|
1021 |
|
1022 The macros :cfunc:`Py_INCREF` and :cfunc:`Py_DECREF` do not check for *NULL* |
|
1023 pointers --- however, their variants :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` |
|
1024 do. |
|
1025 |
|
1026 The macros for checking for a particular object type (``Pytype_Check()``) don't |
|
1027 check for *NULL* pointers --- again, there is much code that calls several of |
|
1028 these in a row to test an object against various different expected types, and |
|
1029 this would generate redundant tests. There are no variants with *NULL* |
|
1030 checking. |
|
1031 |
|
1032 The C function calling mechanism guarantees that the argument list passed to C |
|
1033 functions (``args`` in the examples) is never *NULL* --- in fact it guarantees |
|
1034 that it is always a tuple. [#]_ |
|
1035 |
|
1036 It is a severe error to ever let a *NULL* pointer "escape" to the Python user. |
|
1037 |
|
1038 .. Frank Stajano: |
|
1039 A pedagogically buggy example, along the lines of the previous listing, would |
|
1040 be helpful here -- showing in more concrete terms what sort of actions could |
|
1041 cause the problem. I can't very well imagine it from the description. |
|
1042 |
|
1043 |
|
1044 .. _cplusplus: |
|
1045 |
|
1046 Writing Extensions in C++ |
|
1047 ========================= |
|
1048 |
|
1049 It is possible to write extension modules in C++. Some restrictions apply. If |
|
1050 the main program (the Python interpreter) is compiled and linked by the C |
|
1051 compiler, global or static objects with constructors cannot be used. This is |
|
1052 not a problem if the main program is linked by the C++ compiler. Functions that |
|
1053 will be called by the Python interpreter (in particular, module initialization |
|
1054 functions) have to be declared using ``extern "C"``. It is unnecessary to |
|
1055 enclose the Python header files in ``extern "C" {...}`` --- they use this form |
|
1056 already if the symbol ``__cplusplus`` is defined (all recent C++ compilers |
|
1057 define this symbol). |
|
1058 |
|
1059 |
|
1060 .. _using-cobjects: |
|
1061 |
|
1062 Providing a C API for an Extension Module |
|
1063 ========================================= |
|
1064 |
|
1065 .. sectionauthor:: Konrad Hinsen <hinsen@cnrs-orleans.fr> |
|
1066 |
|
1067 |
|
1068 Many extension modules just provide new functions and types to be used from |
|
1069 Python, but sometimes the code in an extension module can be useful for other |
|
1070 extension modules. For example, an extension module could implement a type |
|
1071 "collection" which works like lists without order. Just like the standard Python |
|
1072 list type has a C API which permits extension modules to create and manipulate |
|
1073 lists, this new collection type should have a set of C functions for direct |
|
1074 manipulation from other extension modules. |
|
1075 |
|
1076 At first sight this seems easy: just write the functions (without declaring them |
|
1077 ``static``, of course), provide an appropriate header file, and document |
|
1078 the C API. And in fact this would work if all extension modules were always |
|
1079 linked statically with the Python interpreter. When modules are used as shared |
|
1080 libraries, however, the symbols defined in one module may not be visible to |
|
1081 another module. The details of visibility depend on the operating system; some |
|
1082 systems use one global namespace for the Python interpreter and all extension |
|
1083 modules (Windows, for example), whereas others require an explicit list of |
|
1084 imported symbols at module link time (AIX is one example), or offer a choice of |
|
1085 different strategies (most Unices). And even if symbols are globally visible, |
|
1086 the module whose functions one wishes to call might not have been loaded yet! |
|
1087 |
|
1088 Portability therefore requires not to make any assumptions about symbol |
|
1089 visibility. This means that all symbols in extension modules should be declared |
|
1090 ``static``, except for the module's initialization function, in order to |
|
1091 avoid name clashes with other extension modules (as discussed in section |
|
1092 :ref:`methodtable`). And it means that symbols that *should* be accessible from |
|
1093 other extension modules must be exported in a different way. |
|
1094 |
|
1095 Python provides a special mechanism to pass C-level information (pointers) from |
|
1096 one extension module to another one: CObjects. A CObject is a Python data type |
|
1097 which stores a pointer (:ctype:`void \*`). CObjects can only be created and |
|
1098 accessed via their C API, but they can be passed around like any other Python |
|
1099 object. In particular, they can be assigned to a name in an extension module's |
|
1100 namespace. Other extension modules can then import this module, retrieve the |
|
1101 value of this name, and then retrieve the pointer from the CObject. |
|
1102 |
|
1103 There are many ways in which CObjects can be used to export the C API of an |
|
1104 extension module. Each name could get its own CObject, or all C API pointers |
|
1105 could be stored in an array whose address is published in a CObject. And the |
|
1106 various tasks of storing and retrieving the pointers can be distributed in |
|
1107 different ways between the module providing the code and the client modules. |
|
1108 |
|
1109 The following example demonstrates an approach that puts most of the burden on |
|
1110 the writer of the exporting module, which is appropriate for commonly used |
|
1111 library modules. It stores all C API pointers (just one in the example!) in an |
|
1112 array of :ctype:`void` pointers which becomes the value of a CObject. The header |
|
1113 file corresponding to the module provides a macro that takes care of importing |
|
1114 the module and retrieving its C API pointers; client modules only have to call |
|
1115 this macro before accessing the C API. |
|
1116 |
|
1117 The exporting module is a modification of the :mod:`spam` module from section |
|
1118 :ref:`extending-simpleexample`. The function :func:`spam.system` does not call |
|
1119 the C library function :cfunc:`system` directly, but a function |
|
1120 :cfunc:`PySpam_System`, which would of course do something more complicated in |
|
1121 reality (such as adding "spam" to every command). This function |
|
1122 :cfunc:`PySpam_System` is also exported to other extension modules. |
|
1123 |
|
1124 The function :cfunc:`PySpam_System` is a plain C function, declared |
|
1125 ``static`` like everything else:: |
|
1126 |
|
1127 static int |
|
1128 PySpam_System(const char *command) |
|
1129 { |
|
1130 return system(command); |
|
1131 } |
|
1132 |
|
1133 The function :cfunc:`spam_system` is modified in a trivial way:: |
|
1134 |
|
1135 static PyObject * |
|
1136 spam_system(PyObject *self, PyObject *args) |
|
1137 { |
|
1138 const char *command; |
|
1139 int sts; |
|
1140 |
|
1141 if (!PyArg_ParseTuple(args, "s", &command)) |
|
1142 return NULL; |
|
1143 sts = PySpam_System(command); |
|
1144 return Py_BuildValue("i", sts); |
|
1145 } |
|
1146 |
|
1147 In the beginning of the module, right after the line :: |
|
1148 |
|
1149 #include "Python.h" |
|
1150 |
|
1151 two more lines must be added:: |
|
1152 |
|
1153 #define SPAM_MODULE |
|
1154 #include "spammodule.h" |
|
1155 |
|
1156 The ``#define`` is used to tell the header file that it is being included in the |
|
1157 exporting module, not a client module. Finally, the module's initialization |
|
1158 function must take care of initializing the C API pointer array:: |
|
1159 |
|
1160 PyMODINIT_FUNC |
|
1161 initspam(void) |
|
1162 { |
|
1163 PyObject *m; |
|
1164 static void *PySpam_API[PySpam_API_pointers]; |
|
1165 PyObject *c_api_object; |
|
1166 |
|
1167 m = Py_InitModule("spam", SpamMethods); |
|
1168 if (m == NULL) |
|
1169 return; |
|
1170 |
|
1171 /* Initialize the C API pointer array */ |
|
1172 PySpam_API[PySpam_System_NUM] = (void *)PySpam_System; |
|
1173 |
|
1174 /* Create a CObject containing the API pointer array's address */ |
|
1175 c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL); |
|
1176 |
|
1177 if (c_api_object != NULL) |
|
1178 PyModule_AddObject(m, "_C_API", c_api_object); |
|
1179 } |
|
1180 |
|
1181 Note that ``PySpam_API`` is declared ``static``; otherwise the pointer |
|
1182 array would disappear when :func:`initspam` terminates! |
|
1183 |
|
1184 The bulk of the work is in the header file :file:`spammodule.h`, which looks |
|
1185 like this:: |
|
1186 |
|
1187 #ifndef Py_SPAMMODULE_H |
|
1188 #define Py_SPAMMODULE_H |
|
1189 #ifdef __cplusplus |
|
1190 extern "C" { |
|
1191 #endif |
|
1192 |
|
1193 /* Header file for spammodule */ |
|
1194 |
|
1195 /* C API functions */ |
|
1196 #define PySpam_System_NUM 0 |
|
1197 #define PySpam_System_RETURN int |
|
1198 #define PySpam_System_PROTO (const char *command) |
|
1199 |
|
1200 /* Total number of C API pointers */ |
|
1201 #define PySpam_API_pointers 1 |
|
1202 |
|
1203 |
|
1204 #ifdef SPAM_MODULE |
|
1205 /* This section is used when compiling spammodule.c */ |
|
1206 |
|
1207 static PySpam_System_RETURN PySpam_System PySpam_System_PROTO; |
|
1208 |
|
1209 #else |
|
1210 /* This section is used in modules that use spammodule's API */ |
|
1211 |
|
1212 static void **PySpam_API; |
|
1213 |
|
1214 #define PySpam_System \ |
|
1215 (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM]) |
|
1216 |
|
1217 /* Return -1 and set exception on error, 0 on success. */ |
|
1218 static int |
|
1219 import_spam(void) |
|
1220 { |
|
1221 PyObject *module = PyImport_ImportModule("spam"); |
|
1222 |
|
1223 if (module != NULL) { |
|
1224 PyObject *c_api_object = PyObject_GetAttrString(module, "_C_API"); |
|
1225 if (c_api_object == NULL) |
|
1226 return -1; |
|
1227 if (PyCObject_Check(c_api_object)) |
|
1228 PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object); |
|
1229 Py_DECREF(c_api_object); |
|
1230 } |
|
1231 return 0; |
|
1232 } |
|
1233 |
|
1234 #endif |
|
1235 |
|
1236 #ifdef __cplusplus |
|
1237 } |
|
1238 #endif |
|
1239 |
|
1240 #endif /* !defined(Py_SPAMMODULE_H) */ |
|
1241 |
|
1242 All that a client module must do in order to have access to the function |
|
1243 :cfunc:`PySpam_System` is to call the function (or rather macro) |
|
1244 :cfunc:`import_spam` in its initialization function:: |
|
1245 |
|
1246 PyMODINIT_FUNC |
|
1247 initclient(void) |
|
1248 { |
|
1249 PyObject *m; |
|
1250 |
|
1251 m = Py_InitModule("client", ClientMethods); |
|
1252 if (m == NULL) |
|
1253 return; |
|
1254 if (import_spam() < 0) |
|
1255 return; |
|
1256 /* additional initialization can happen here */ |
|
1257 } |
|
1258 |
|
1259 The main disadvantage of this approach is that the file :file:`spammodule.h` is |
|
1260 rather complicated. However, the basic structure is the same for each function |
|
1261 that is exported, so it has to be learned only once. |
|
1262 |
|
1263 Finally it should be mentioned that CObjects offer additional functionality, |
|
1264 which is especially useful for memory allocation and deallocation of the pointer |
|
1265 stored in a CObject. The details are described in the Python/C API Reference |
|
1266 Manual in the section :ref:`cobjects` and in the implementation of CObjects (files |
|
1267 :file:`Include/cobject.h` and :file:`Objects/cobject.c` in the Python source |
|
1268 code distribution). |
|
1269 |
|
1270 .. rubric:: Footnotes |
|
1271 |
|
1272 .. [#] An interface for this function already exists in the standard module :mod:`os` |
|
1273 --- it was chosen as a simple and straightforward example. |
|
1274 |
|
1275 .. [#] The metaphor of "borrowing" a reference is not completely correct: the owner |
|
1276 still has a copy of the reference. |
|
1277 |
|
1278 .. [#] Checking that the reference count is at least 1 **does not work** --- the |
|
1279 reference count itself could be in freed memory and may thus be reused for |
|
1280 another object! |
|
1281 |
|
1282 .. [#] These guarantees don't hold when you use the "old" style calling convention --- |
|
1283 this is still found in much existing code. |
|
1284 |