% Copyright (c) 2009 Nokia Corporation
%
% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at
%
%     http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.

\chapter{Extending and Embedding PyS60}
\label{extendandembed}
Extending and Embedding Python is explained in the following sections with examples
wherever necessary.

\section{Extending PyS60}
\label{sec:extendingPyS60}

The general rules and guidelines for writing Python extensions apply
in the S60 Python environment as well.

The steps for the implementation of an extension modules include:

\begin{itemize}
\item Preparation of the data structures that make the C/C++ coded extensions
      visible to the Python interpreter and make it possible to perform calls from
      Python to C/C++ code

\item Conversions between C/C++ representations of the Python objects and
      object types used in the extension code

\item Maintenance of the reference counts of the C/C++ representations of the
      Python objects

\item Passing of exceptions between C/C++ code and Python

\item Management of interpreter's thread state and the interpreter lock
\end{itemize}

In addition to the concerns common for all Python C extensions, the
following principles should be considered when implementing new Python
interfaces in the S60 environment:

\begin{itemize}
\item Maximize the usage of Python's built-in types at the interfaces.

\item Related to the above: design interfaces in such a way that information
      can be passed between them with minimal conversions.

\item Convert Symbian operating system exceptions or errors to Python exceptions.

\item Unicode strings are used at the interfaces to represent text that gets
      shown on the GUI. They can be passed to and from Symbian operating system
      without conversions.

\item While performing potentially long-lasting or blocking calls from an
      extension implementation to services outside the interpreter, the
      interpreter lock must be released and then re-acquired after the call.

\item Rather than always implementing a thin wrapper on top of a Symbian OS
      facility, consider the actual task for which the script writer needs the
      particular interface. For example, if the task involves interaction with
      the users using the GUI, the script writer's interest may well be limited
      to performing the interaction or information exchange in a way that is
      compatible with the UI style rather than having full control of the
      low-level details of the GUI implementation.

\item The C/C++ implementation of a Python interface should be optimized for
      performance and covering access to the necessary features of the
      underlying Platform. Where necessary, the Python programming interface can
      be further refined by wrapper modules written in Python.
\end{itemize}

The pyd name should be of the format 'kf_<module-name>.pyd'. Note that this change
is required only for the pyd name and module name is not required to have this prefix.

The module initialization function must be exported at ordinal 1. The module
identification is based on the filename only. As a special feature of PyS60, an
optional module finalizer function may be exported at ordinal 2.

The extension modules added by the developer should be placed in the module repo
folder of the PyS60 Application Packager so that the compiled PYD's can be picked
up while packaging. For information on this topic refer \ref{subsec:distributeextmodules}, Distributing Extension Modules.

\subsection{Example Extension Module}
To demonstrate the writing of an extension module the source and MMP contents
for an example module , 'elemlist' is given below. This module is about extracting
the pointers in a cons cell. A cons cell is composed of two pointers. The car and cdr
are primitive operations upon linked lists composed of cons cells. The car operation
extracts the first pointer, and the cdr operation extracts the second.

\begin{verbatim}
#include "Python.h"

/* type-definition & utility-macros */
typedef struct {
    PyObject_HEAD
    PyObject *car, *cdr;
} cons_cell;

staticforward PyTypeObject cons_type;

/* a typetesting macro (we don't use it here) */
#define is_cons(v) ((v)->ob_type == &cons_type)

/* macros to access car & cdr, both as lvalues & rvalues */
#define carof(v) (((cons_cell*)(v))->car)
#define cdrof(v) (((cons_cell*)(v))->cdr)

/* ctor (factory-function) and dtor */
static cons_cell*
cons_new(PyObject *car, PyObject *cdr)
{
    cons_cell *cons = PyObject_NEW(cons_cell, &cons_type);
    if(cons) {
        cons->car = car; Py_INCREF(car);  /* INCREF when holding a PyObject* */
        cons->cdr = cdr; Py_INCREF(cdr);  /* ditto */
    }
    return cons;
}
static void
cons_dealloc(cons_cell* cons)
{
    /* DECREF when releasing previously-held PyObject*'s */
    Py_DECREF(carof(cons)); Py_DECREF(cdrof(cons));
    PyObject_DEL(cons);
}

/* Python type-object */
statichere PyTypeObject cons_type = {
    PyObject_HEAD_INIT(0)    /* initialize to 0 to ensure Win32 portability */
    0,                 /*ob_size*/
    "cons",            /*tp_name*/
    sizeof(cons_cell), /*tp_basicsize*/
    0,                 /*tp_itemsize*/
    /* methods */
    (destructor)cons_dealloc, /*tp_dealloc*/
    /* implied by ISO C: all zeros thereafter */
};

/* module-functions */
static PyObject*
cons(PyObject *self, PyObject *args)    /* the exposed factory-function */
{
    PyObject *car, *cdr;
    if(!PyArg_ParseTuple(args, "OO", &car, &cdr))
        return 0;
    return (PyObject*)cons_new(car, cdr);
}
static PyObject*
car(PyObject *self, PyObject *args)     /* car-accessor */
{
    PyObject *cons;
    if(!PyArg_ParseTuple(args, "O!", &cons_type, &cons))  /* type-checked */
        return 0;
    return Py_BuildValue("O", carof(cons));
}
static PyObject*
cdr(PyObject *self, PyObject *args)     /* cdr-accessor */
{
    PyObject *cons;
    if(!PyArg_ParseTuple(args, "O!", &cons_type, &cons))  /* type-checked */
        return 0;
    return Py_BuildValue("O", cdrof(cons));
}
static PyMethodDef elemlist_methods[] = {
    {"cons",   cons,   METH_VARARGS},
    {"car",    car,    METH_VARARGS},
    {"cdr",    cdr,    METH_VARARGS},
    {0, 0}
};

/* module entry-point (module-initialization) function */
PyMODINIT_FUNC
initelemlist(void)
{
    /* Create the module and add the functions */
    PyObject *m = Py_InitModule("elemlist", elemlist_methods);
    /* Finish initializing the type-objects */
    cons_type.ob_type = &PyType_Type;
}
\end{verbatim}
A brief description of the concepts involved in writing an extension module is
put down below. For more information about writing an extension module refer
\citetitle[../ext/intro.html]{Extending Python} from the mainline Python documentation.

The header file Python.h makes the Python API's accessible in the code.
\cfunction{car()}, \cfunction{cdr()} and \cfunction{cons()} are the functions exposed
at Python level so that they can be called after importing the \code{elemlist} module.
The C implementation of these functions take arguments as Python objects.
To do anything with them in the C function we have to convert them to C values.

\code{elemlist_methods} is the method table for the module. The method table is
passed to the interpreter in the module's initialization function, \cfunction{initelemlist()}.
The initialization function must be named \cfunction{initname()}, where name is the
name of the module, and should be the only non-static item defined in the module file.
The PyMODINIT_FUNC declares the function as \code{void} return type, declares any special
linkage declarations required by the platform, and for C++ declares the function
as \code{extern "C"}. When the Python program imports module  elemlist for the
first time, \cfunction{initelemlist()} is called.

The MMP file contents for the above source is as follows:

\begin{verbatim}
TARGETTYPE  dll
TARGET    kf_elemlist.pyd

CAPABILITY LocalServices NetworkServices ReadUserData WriteUserData UserEnvironment

NOSTRICTDEF
DEFFILE   elemlist.def

/* If global data is present in the extension module then this macro should be
 * defined in the mmp file.
 */
EPOCALLOWDLLDATA

SYSTEMINCLUDE    \epoc32\include\python25
SYSTEMINCLUDE    \epoc32\include\stdapis
SYSTEMINCLUDE    \epoc32\include

LIBRARY     python25.lib

SOURCEPATH  ..\src
SOURCE      elemlist.c
\end{verbatim}


An example usage of the 'elemlist' extension module can be:

\begin{verbatim}
from elemlist import *
cell = cons(1, 2)
print "car(cell) :", car(cell)
print "cdr(cell) :", cdr(cell)
\end{verbatim}

\subsection{Compiling the extension module}

{\bf Requirements} \break
\begin{itemize}
\item Series 60 SDK, 3rdEd or higher
\item Python for S60 1.9.x SDK package
\item Open C/C++ plug-in. Refer the release notes for the version of OpenC to be
      installed for this release.
\end{itemize}

{\bf Installation} \break
Place the Python for Series 60 SDK 3rdEd under the Symbian SDK installation
directory, at the same level as the "epoc32" directory. Extract the SDK zip
package here(On Windows if you have WinZip installed, right-click on the zip
and in the menu select : 'Winzip->Extract to here')

{\bf Building} \break
Modify the mmp file to include "Location" capability while building for 3rdEdFp2
and higher devices. A script file build_all.cmd that does all the necessary
steps (and some extra cleanup, just to be sure) has been provided for
convenience. You can either use that or perform the build manually using these
instructions.

\begin{itemize}
\item Go to the elemlist directory. Enter:\\ {\bf bldmake bldfiles}

\item To build the extension for the device, enter: \\
{\bf abld build gcce urel\\
     abld freeze gcce\\
     abld build gcce urel\\}

You should find the built module in (path to your SDK)$\backslash$epoc32$\backslash$release$\backslash$gcce$\backslash$urel$\backslash$kf_elemlist.pyd

\item To build it for the emulator environment, enter: \\
{\bf abld build winscw udeb\\
     abld freeze winscw\\
     abld build winscw udeb\\}

You should find the built module in (path to your SDK)$\backslash$epoc32$\backslash$release$\backslash$winscw$\backslash$ubed$\backslash$kf_elemlist.pyd
\end{itemize}

\note {The "freeze" step needs to be done only when you add any function exports.
       After "freeze", just one "abld build gcce urel" or "abld build winscw udeb"
       will rebuild the code properly.}

\subsection{Distributing extension modules}
\label{subsec:distributeextmodules}
Distributing extension modules to application developers is much easier now with
the PyS60 application packager scanning for dependencies automatically. Please
refer the topic {\bf Distributing extension modules to application developers}
in the Chapter \citetitle[./modulerepo.html]{Module Repository} for information on how this is done.

\section{Embedding PyS60}
\label{sec:embedPyS60}
There is not much change with respect to embedding Python from what is mentioned in the
Python mainline document apart from the custom memory allocator which is explained
later in this Section.

The following code snippet, which prints 'Hello World!' on the screen,
demonstrates the embedding of Python interpreter in a C code:

The source file contents are as follows:
\begin{verbatim}
#include <Python.h>
/* This is a GCCE toolchain workaround needed when compiling with GCCE
   and using main() entry point */
#ifdef __GCCE__
#include <staticlibinit_gcce.h>
#endif

int main(void)
{

    SPy_DLC_Init();
    SPy_SetAllocator(SPy_DLC_Alloc, SPy_DLC_Realloc, SPy_DLC_Free, NULL);
    Py_Initialize();
    PyRun_SimpleString("print 'Hello World!'");
    Py_Finalize();
    SPy_DLC_Fini();
    return 0;
}
\end{verbatim}

The basic initialization function is \cfunction{Py_Initialize()}. This initializes
the table of loaded modules, and creates the fundamental modules __builtin__,
__main__ and sys. It also initializes the module search path (sys.path).
\cfunction{Py_Finalize()} is called when the application is done with its use of
Python and wants to free all memory allocated by Python.

PyS60 provides a DLC custom allocator which can be used instead of Python memory allocator.
\cfunction{SPy_SetAllocator()} is used for redirecting the allocator used by
Python. The arguments to this function are the custom functions for allocating,
reallocating, freeing the memory and a context pointer in that order. \cfunction{SPy_DLC_Init()}
is used for initializing the DLC custom allocator. \cfunction{SPy_DLC_Fini()} is used
for finalizing the DLC custom allocator and doing a memory cleanup. If you want to use
your own custom allocator you will have to define the allocation, reallocation
and free memory functions and pass the function names to \cfunction{SPy_SetAllocator()}.

For more information on embedding Python, refer \citetitle[../ext/embedding.html]{Embedding Python in Another Application}

The MMP file contents for the above source is as follows:

\begin{verbatim}
TARGET        helloworld.exe
TARGETTYPE    exe

SYSTEMINCLUDE    \epoc32\include\python25
SYSTEMINCLUDE    \epoc32\include\stdapis
SYSTEMINCLUDE    \epoc32\include

/* Using main() as entry point */
STATICLIBRARY libcrt0.lib

/* libc and euser are always needed when using main() entry point */
LIBRARY libc.lib
LIBRARY euser.lib
LIBRARY python25.lib

SOURCEPATH      ..\src
SOURCE          helloworld.cpp

\end{verbatim}

\section{Porting 1.4.x to 1.9.x}
\label{sec:porting}
The changes needed for porting existing native PyS60 extensions are as follows:

\begin{itemize}
\item From Symbian 9.1 onwards Symbian allows Writable Static Data in a DLL
      by making use of EPOCALLOWDLLDATA keyword in the mmp file. Main reason for
      this one is for porting some non-Symbian applications onto Symbian. Thus
      TLS functionality is no longer needed. Use EPOCALLOWDLLDATA in the MMP file
      if the module has initialized static data.

\item Use \cfunction{PyGILState_Ensure()} and \cfunction{PyGILState_Release()} functions for acquiring and
      releasing the global interpreter lock, instead of using
      \cfunction{PyEval_RestoreThread(PYTHON_TLS->thread_state)} and
      \cfunction{PyEval_SaveThread()}.

\item The interpreter DLL name is changed to python25.lib. This change has to be reflected in the MMP file
      so that the module is linked against this DLL instead of python222.lib used in 1.4.x.

\item The Python header files are now in $\backslash$epoc32$\backslash$include$\backslash$Python25 and hence
      the MMP file needs to be updated accordingly.
\item The pyd name should be kf_<module-name>.pyd.\\
\note {This change is required only for the pyd name and module name is not required to have this prefix}.

\item Packaging an extension module is explained in section \ref{subsec:distributeextmodules}, Distributing extension modules.

\note{The init-function still needs to be exported in the pyd at ordinal 1.}
\end{itemize}

{\bf Script related changes} \break

\begin{itemize}
\item The main script of the PyS60 applications, default.py is not executed directly,
      as was the case in PyS60 1.4.x. The wrapper script, launcher.py is first
      executed which in turn does an execfile on the default.py. Therefore, to exit
      the application programmatically use \cfunction{appuifw.app.set_exit()} or
      \cfunction{sys.exit()}

\item PyS60 1.4.x extension modules socket and calendar are renamed to btsocket and
      e32calendar due to the conflicting names with Python core modules. Two
      packaging modes pys60 and pycore have been provided with Application Packager
      tool to maintain the compatibility with PyS60 1.4.x binaries. The existing scripts
      dependent on these extension modules need not be modified if it is packaged
      with pys60 mode.

\item Unlike PyS60 1.4.x the module names are case sensitive from PyS60 1.9.x. So
      the scripts written for 1.4.x will require changes to account for this.
\end{itemize}
