--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/symbian-qemu-0.9.1-12/python-2.6.1/Doc/library/multiprocessing.rst Fri Jul 31 15:01:17 2009 +0100
@@ -0,0 +1,2128 @@
+:mod:`multiprocessing` --- Process-based "threading" interface
+==============================================================
+
+.. module:: multiprocessing
+ :synopsis: Process-based "threading" interface.
+
+.. versionadded:: 2.6
+
+
+Introduction
+----------------------
+
+:mod:`multiprocessing` is a package that supports spawning processes using an
+API similar to the :mod:`threading` module. The :mod:`multiprocessing` package
+offers both local and remote concurrency, effectively side-stepping the
+:term:`Global Interpreter Lock` by using subprocesses instead of threads. Due
+to this, the :mod:`multiprocessing` module allows the programmer to fully
+leverage multiple processors on a given machine. It runs on both Unix and
+Windows.
+
+.. warning::
+
+ Some of this package's functionality requires a functioning shared semaphore
+ implementation on the host operating system. Without one, the
+ :mod:`multiprocessing.synchronize` module will be disabled, and attempts to
+ import it will result in an :exc:`ImportError`. See
+ :issue:`3770` for additional information.
+
+.. note::
+
+ Functionality within this package requires that the ``__main__`` method be
+ importable by the children. This is covered in :ref:`multiprocessing-programming`
+ however it is worth pointing out here. This means that some examples, such
+ as the :class:`multiprocessing.Pool` examples will not work in the
+ interactive interpreter. For example::
+
+ >>> from multiprocessing import Pool
+ >>> p = Pool(5)
+ >>> def f(x):
+ ... return x*x
+ ...
+ >>> p.map(f, [1,2,3])
+ Process PoolWorker-1:
+ Process PoolWorker-2:
+ Traceback (most recent call last):
+ Traceback (most recent call last):
+ AttributeError: 'module' object has no attribute 'f'
+ AttributeError: 'module' object has no attribute 'f'
+ AttributeError: 'module' object has no attribute 'f'
+
+
+The :class:`Process` class
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In :mod:`multiprocessing`, processes are spawned by creating a :class:`Process`
+object and then calling its :meth:`~Process.start` method. :class:`Process`
+follows the API of :class:`threading.Thread`. A trivial example of a
+multiprocess program is ::
+
+ from multiprocessing import Process
+
+ def f(name):
+ print 'hello', name
+
+ if __name__ == '__main__':
+ p = Process(target=f, args=('bob',))
+ p.start()
+ p.join()
+
+To show the individual process IDs involved, here is an expanded example::
+
+ from multiprocessing import Process
+ import os
+
+ def info(title):
+ print title
+ print 'module name:', __name__
+ print 'parent process:', os.getppid()
+ print 'process id:', os.getpid()
+
+ def f(name):
+ info('function f')
+ print 'hello', name
+
+ if __name__ == '__main__':
+ info('main line')
+ p = Process(target=f, args=('bob',))
+ p.start()
+ p.join()
+
+For an explanation of why (on Windows) the ``if __name__ == '__main__'`` part is
+necessary, see :ref:`multiprocessing-programming`.
+
+
+
+Exchanging objects between processes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:mod:`multiprocessing` supports two types of communication channel between
+processes:
+
+**Queues**
+
+ The :class:`Queue` class is a near clone of :class:`Queue.Queue`. For
+ example::
+
+ from multiprocessing import Process, Queue
+
+ def f(q):
+ q.put([42, None, 'hello'])
+
+ if __name__ == '__main__':
+ q = Queue()
+ p = Process(target=f, args=(q,))
+ p.start()
+ print q.get() # prints "[42, None, 'hello']"
+ p.join()
+
+ Queues are thread and process safe.
+
+**Pipes**
+
+ The :func:`Pipe` function returns a pair of connection objects connected by a
+ pipe which by default is duplex (two-way). For example::
+
+ from multiprocessing import Process, Pipe
+
+ def f(conn):
+ conn.send([42, None, 'hello'])
+ conn.close()
+
+ if __name__ == '__main__':
+ parent_conn, child_conn = Pipe()
+ p = Process(target=f, args=(child_conn,))
+ p.start()
+ print parent_conn.recv() # prints "[42, None, 'hello']"
+ p.join()
+
+ The two connection objects returned by :func:`Pipe` represent the two ends of
+ the pipe. Each connection object has :meth:`~Connection.send` and
+ :meth:`~Connection.recv` methods (among others). Note that data in a pipe
+ may become corrupted if two processes (or threads) try to read from or write
+ to the *same* end of the pipe at the same time. Of course there is no risk
+ of corruption from processes using different ends of the pipe at the same
+ time.
+
+
+Synchronization between processes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:mod:`multiprocessing` contains equivalents of all the synchronization
+primitives from :mod:`threading`. For instance one can use a lock to ensure
+that only one process prints to standard output at a time::
+
+ from multiprocessing import Process, Lock
+
+ def f(l, i):
+ l.acquire()
+ print 'hello world', i
+ l.release()
+
+ if __name__ == '__main__':
+ lock = Lock()
+
+ for num in range(10):
+ Process(target=f, args=(lock, num)).start()
+
+Without using the lock output from the different processes is liable to get all
+mixed up.
+
+
+Sharing state between processes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As mentioned above, when doing concurrent programming it is usually best to
+avoid using shared state as far as possible. This is particularly true when
+using multiple processes.
+
+However, if you really do need to use some shared data then
+:mod:`multiprocessing` provides a couple of ways of doing so.
+
+**Shared memory**
+
+ Data can be stored in a shared memory map using :class:`Value` or
+ :class:`Array`. For example, the following code ::
+
+ from multiprocessing import Process, Value, Array
+
+ def f(n, a):
+ n.value = 3.1415927
+ for i in range(len(a)):
+ a[i] = -a[i]
+
+ if __name__ == '__main__':
+ num = Value('d', 0.0)
+ arr = Array('i', range(10))
+
+ p = Process(target=f, args=(num, arr))
+ p.start()
+ p.join()
+
+ print num.value
+ print arr[:]
+
+ will print ::
+
+ 3.1415927
+ [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
+
+ The ``'d'`` and ``'i'`` arguments used when creating ``num`` and ``arr`` are
+ typecodes of the kind used by the :mod:`array` module: ``'d'`` indicates a
+ double precision float and ``'i'`` indicates a signed integer. These shared
+ objects will be process and thread safe.
+
+ For more flexibility in using shared memory one can use the
+ :mod:`multiprocessing.sharedctypes` module which supports the creation of
+ arbitrary ctypes objects allocated from shared memory.
+
+**Server process**
+
+ A manager object returned by :func:`Manager` controls a server process which
+ holds Python objects and allows other processes to manipulate them using
+ proxies.
+
+ A manager returned by :func:`Manager` will support types :class:`list`,
+ :class:`dict`, :class:`Namespace`, :class:`Lock`, :class:`RLock`,
+ :class:`Semaphore`, :class:`BoundedSemaphore`, :class:`Condition`,
+ :class:`Event`, :class:`Queue`, :class:`Value` and :class:`Array`. For
+ example, ::
+
+ from multiprocessing import Process, Manager
+
+ def f(d, l):
+ d[1] = '1'
+ d['2'] = 2
+ d[0.25] = None
+ l.reverse()
+
+ if __name__ == '__main__':
+ manager = Manager()
+
+ d = manager.dict()
+ l = manager.list(range(10))
+
+ p = Process(target=f, args=(d, l))
+ p.start()
+ p.join()
+
+ print d
+ print l
+
+ will print ::
+
+ {0.25: None, 1: '1', '2': 2}
+ [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
+
+ Server process managers are more flexible than using shared memory objects
+ because they can be made to support arbitrary object types. Also, a single
+ manager can be shared by processes on different computers over a network.
+ They are, however, slower than using shared memory.
+
+
+Using a pool of workers
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :class:`~multiprocessing.pool.Pool` class represents a pool of worker
+processes. It has methods which allows tasks to be offloaded to the worker
+processes in a few different ways.
+
+For example::
+
+ from multiprocessing import Pool
+
+ def f(x):
+ return x*x
+
+ if __name__ == '__main__':
+ pool = Pool(processes=4) # start 4 worker processes
+ result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously
+ print result.get(timeout=1) # prints "100" unless your computer is *very* slow
+ print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
+
+
+Reference
+---------
+
+The :mod:`multiprocessing` package mostly replicates the API of the
+:mod:`threading` module.
+
+
+:class:`Process` and exceptions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. class:: Process([group[, target[, name[, args[, kwargs]]]]])
+
+ Process objects represent activity that is run in a separate process. The
+ :class:`Process` class has equivalents of all the methods of
+ :class:`threading.Thread`.
+
+ The constructor should always be called with keyword arguments. *group*
+ should always be ``None``; it exists solely for compatibility with
+ :class:`threading.Thread`. *target* is the callable object to be invoked by
+ the :meth:`run()` method. It defaults to ``None``, meaning nothing is
+ called. *name* is the process name. By default, a unique name is constructed
+ of the form 'Process-N\ :sub:`1`:N\ :sub:`2`:...:N\ :sub:`k`' where N\
+ :sub:`1`,N\ :sub:`2`,...,N\ :sub:`k` is a sequence of integers whose length
+ is determined by the *generation* of the process. *args* is the argument
+ tuple for the target invocation. *kwargs* is a dictionary of keyword
+ arguments for the target invocation. By default, no arguments are passed to
+ *target*.
+
+ If a subclass overrides the constructor, it must make sure it invokes the
+ base class constructor (:meth:`Process.__init__`) before doing anything else
+ to the process.
+
+ .. method:: run()
+
+ Method representing the process's activity.
+
+ You may override this method in a subclass. The standard :meth:`run`
+ method invokes the callable object passed to the object's constructor as
+ the target argument, if any, with sequential and keyword arguments taken
+ from the *args* and *kwargs* arguments, respectively.
+
+ .. method:: start()
+
+ Start the process's activity.
+
+ This must be called at most once per process object. It arranges for the
+ object's :meth:`run` method to be invoked in a separate process.
+
+ .. method:: join([timeout])
+
+ Block the calling thread until the process whose :meth:`join` method is
+ called terminates or until the optional timeout occurs.
+
+ If *timeout* is ``None`` then there is no timeout.
+
+ A process can be joined many times.
+
+ A process cannot join itself because this would cause a deadlock. It is
+ an error to attempt to join a process before it has been started.
+
+ .. attribute:: name
+
+ The process's name.
+
+ The name is a string used for identification purposes only. It has no
+ semantics. Multiple processes may be given the same name. The initial
+ name is set by the constructor.
+
+ .. method:: is_alive
+
+ Return whether the process is alive.
+
+ Roughly, a process object is alive from the moment the :meth:`start`
+ method returns until the child process terminates.
+
+ .. attribute:: daemon
+
+ The process's daemon flag, a Boolean value. This must be called before
+ :meth:`start` is called.
+
+ The initial value is inherited from the creating process.
+
+ When a process exits, it attempts to terminate all of its daemonic child
+ processes.
+
+ Note that a daemonic process is not allowed to create child processes.
+ Otherwise a daemonic process would leave its children orphaned if it gets
+ terminated when its parent process exits.
+
+ In addition to the :class:`Threading.Thread` API, :class:`Process` objects
+ also support the following attributes and methods:
+
+ .. attribute:: pid
+
+ Return the process ID. Before the process is spawned, this will be
+ ``None``.
+
+ .. attribute:: exitcode
+
+ The child's exit code. This will be ``None`` if the process has not yet
+ terminated. A negative value *-N* indicates that the child was terminated
+ by signal *N*.
+
+ .. attribute:: authkey
+
+ The process's authentication key (a byte string).
+
+ When :mod:`multiprocessing` is initialized the main process is assigned a
+ random string using :func:`os.random`.
+
+ When a :class:`Process` object is created, it will inherit the
+ authentication key of its parent process, although this may be changed by
+ setting :attr:`authkey` to another byte string.
+
+ See :ref:`multiprocessing-auth-keys`.
+
+ .. method:: terminate()
+
+ Terminate the process. On Unix this is done using the ``SIGTERM`` signal;
+ on Windows :cfunc:`TerminateProcess` is used. Note that exit handlers and
+ finally clauses, etc., will not be executed.
+
+ Note that descendant processes of the process will *not* be terminated --
+ they will simply become orphaned.
+
+ .. warning::
+
+ If this method is used when the associated process is using a pipe or
+ queue then the pipe or queue is liable to become corrupted and may
+ become unusable by other process. Similarly, if the process has
+ acquired a lock or semaphore etc. then terminating it is liable to
+ cause other processes to deadlock.
+
+ Note that the :meth:`start`, :meth:`join`, :meth:`is_alive` and
+ :attr:`exit_code` methods should only be called by the process that created
+ the process object.
+
+ Example usage of some of the methods of :class:`Process`::
+
+ >>> import multiprocessing, time, signal
+ >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
+ >>> print p, p.is_alive()
+ <Process(Process-1, initial)> False
+ >>> p.start()
+ >>> print p, p.is_alive()
+ <Process(Process-1, started)> True
+ >>> p.terminate()
+ >>> print p, p.is_alive()
+ <Process(Process-1, stopped[SIGTERM])> False
+ >>> p.exitcode == -signal.SIGTERM
+ True
+
+
+.. exception:: BufferTooShort
+
+ Exception raised by :meth:`Connection.recv_bytes_into()` when the supplied
+ buffer object is too small for the message read.
+
+ If ``e`` is an instance of :exc:`BufferTooShort` then ``e.args[0]`` will give
+ the message as a byte string.
+
+
+Pipes and Queues
+~~~~~~~~~~~~~~~~
+
+When using multiple processes, one generally uses message passing for
+communication between processes and avoids having to use any synchronization
+primitives like locks.
+
+For passing messages one can use :func:`Pipe` (for a connection between two
+processes) or a queue (which allows multiple producers and consumers).
+
+The :class:`Queue` and :class:`JoinableQueue` types are multi-producer,
+multi-consumer FIFO queues modelled on the :class:`Queue.Queue` class in the
+standard library. They differ in that :class:`Queue` lacks the
+:meth:`~Queue.Queue.task_done` and :meth:`~Queue.Queue.join` methods introduced
+into Python 2.5's :class:`Queue.Queue` class.
+
+If you use :class:`JoinableQueue` then you **must** call
+:meth:`JoinableQueue.task_done` for each task removed from the queue or else the
+semaphore used to count the number of unfinished tasks may eventually overflow
+raising an exception.
+
+Note that one can also create a shared queue by using a manager object -- see
+:ref:`multiprocessing-managers`.
+
+.. note::
+
+ :mod:`multiprocessing` uses the usual :exc:`Queue.Empty` and
+ :exc:`Queue.Full` exceptions to signal a timeout. They are not available in
+ the :mod:`multiprocessing` namespace so you need to import them from
+ :mod:`Queue`.
+
+
+.. warning::
+
+ If a process is killed using :meth:`Process.terminate` or :func:`os.kill`
+ while it is trying to use a :class:`Queue`, then the data in the queue is
+ likely to become corrupted. This may cause any other processes to get an
+ exception when it tries to use the queue later on.
+
+.. warning::
+
+ As mentioned above, if a child process has put items on a queue (and it has
+ not used :meth:`JoinableQueue.cancel_join_thread`), then that process will
+ not terminate until all buffered items have been flushed to the pipe.
+
+ This means that if you try joining that process you may get a deadlock unless
+ you are sure that all items which have been put on the queue have been
+ consumed. Similarly, if the child process is non-daemonic then the parent
+ process may hang on exit when it tries to join all its non-daemonic children.
+
+ Note that a queue created using a manager does not have this issue. See
+ :ref:`multiprocessing-programming`.
+
+For an example of the usage of queues for interprocess communication see
+:ref:`multiprocessing-examples`.
+
+
+.. function:: Pipe([duplex])
+
+ Returns a pair ``(conn1, conn2)`` of :class:`Connection` objects representing
+ the ends of a pipe.
+
+ If *duplex* is ``True`` (the default) then the pipe is bidirectional. If
+ *duplex* is ``False`` then the pipe is unidirectional: ``conn1`` can only be
+ used for receiving messages and ``conn2`` can only be used for sending
+ messages.
+
+
+.. class:: Queue([maxsize])
+
+ Returns a process shared queue implemented using a pipe and a few
+ locks/semaphores. When a process first puts an item on the queue a feeder
+ thread is started which transfers objects from a buffer into the pipe.
+
+ The usual :exc:`Queue.Empty` and :exc:`Queue.Full` exceptions from the
+ standard library's :mod:`Queue` module are raised to signal timeouts.
+
+ :class:`Queue` implements all the methods of :class:`Queue.Queue` except for
+ :meth:`~Queue.Queue.task_done` and :meth:`~Queue.Queue.join`.
+
+ .. method:: qsize()
+
+ Return the approximate size of the queue. Because of
+ multithreading/multiprocessing semantics, this number is not reliable.
+
+ Note that this may raise :exc:`NotImplementedError` on Unix platforms like
+ Mac OS X where ``sem_getvalue()`` is not implemented.
+
+ .. method:: empty()
+
+ Return ``True`` if the queue is empty, ``False`` otherwise. Because of
+ multithreading/multiprocessing semantics, this is not reliable.
+
+ .. method:: full()
+
+ Return ``True`` if the queue is full, ``False`` otherwise. Because of
+ multithreading/multiprocessing semantics, this is not reliable.
+
+ .. method:: put(item[, block[, timeout]])
+
+ Put item into the queue. If the optional argument *block* is ``True``
+ (the default) and *timeout* is ``None`` (the default), block if necessary until
+ a free slot is available. If *timeout* is a positive number, it blocks at
+ most *timeout* seconds and raises the :exc:`Queue.Full` exception if no
+ free slot was available within that time. Otherwise (*block* is
+ ``False``), put an item on the queue if a free slot is immediately
+ available, else raise the :exc:`Queue.Full` exception (*timeout* is
+ ignored in that case).
+
+ .. method:: put_nowait(item)
+
+ Equivalent to ``put(item, False)``.
+
+ .. method:: get([block[, timeout]])
+
+ Remove and return an item from the queue. If optional args *block* is
+ ``True`` (the default) and *timeout* is ``None`` (the default), block if
+ necessary until an item is available. If *timeout* is a positive number,
+ it blocks at most *timeout* seconds and raises the :exc:`Queue.Empty`
+ exception if no item was available within that time. Otherwise (block is
+ ``False``), return an item if one is immediately available, else raise the
+ :exc:`Queue.Empty` exception (*timeout* is ignored in that case).
+
+ .. method:: get_nowait()
+ get_no_wait()
+
+ Equivalent to ``get(False)``.
+
+ :class:`multiprocessing.Queue` has a few additional methods not found in
+ :class:`Queue.Queue`. These methods are usually unnecessary for most
+ code:
+
+ .. method:: close()
+
+ Indicate that no more data will be put on this queue by the current
+ process. The background thread will quit once it has flushed all buffered
+ data to the pipe. This is called automatically when the queue is garbage
+ collected.
+
+ .. method:: join_thread()
+
+ Join the background thread. This can only be used after :meth:`close` has
+ been called. It blocks until the background thread exits, ensuring that
+ all data in the buffer has been flushed to the pipe.
+
+ By default if a process is not the creator of the queue then on exit it
+ will attempt to join the queue's background thread. The process can call
+ :meth:`cancel_join_thread` to make :meth:`join_thread` do nothing.
+
+ .. method:: cancel_join_thread()
+
+ Prevent :meth:`join_thread` from blocking. In particular, this prevents
+ the background thread from being joined automatically when the process
+ exits -- see :meth:`join_thread`.
+
+
+.. class:: JoinableQueue([maxsize])
+
+ :class:`JoinableQueue`, a :class:`Queue` subclass, is a queue which
+ additionally has :meth:`task_done` and :meth:`join` methods.
+
+ .. method:: task_done()
+
+ Indicate that a formerly enqueued task is complete. Used by queue consumer
+ threads. For each :meth:`~Queue.get` used to fetch a task, a subsequent
+ call to :meth:`task_done` tells the queue that the processing on the task
+ is complete.
+
+ If a :meth:`~Queue.join` is currently blocking, it will resume when all
+ items have been processed (meaning that a :meth:`task_done` call was
+ received for every item that had been :meth:`~Queue.put` into the queue).
+
+ Raises a :exc:`ValueError` if called more times than there were items
+ placed in the queue.
+
+
+ .. method:: join()
+
+ Block until all items in the queue have been gotten and processed.
+
+ The count of unfinished tasks goes up whenever an item is added to the
+ queue. The count goes down whenever a consumer thread calls
+ :meth:`task_done` to indicate that the item was retrieved and all work on
+ it is complete. When the count of unfinished tasks drops to zero,
+ :meth:`~Queue.join` unblocks.
+
+
+Miscellaneous
+~~~~~~~~~~~~~
+
+.. function:: active_children()
+
+ Return list of all live children of the current process.
+
+ Calling this has the side affect of "joining" any processes which have
+ already finished.
+
+.. function:: cpu_count()
+
+ Return the number of CPUs in the system. May raise
+ :exc:`NotImplementedError`.
+
+.. function:: current_process()
+
+ Return the :class:`Process` object corresponding to the current process.
+
+ An analogue of :func:`threading.current_thread`.
+
+.. function:: freeze_support()
+
+ Add support for when a program which uses :mod:`multiprocessing` has been
+ frozen to produce a Windows executable. (Has been tested with **py2exe**,
+ **PyInstaller** and **cx_Freeze**.)
+
+ One needs to call this function straight after the ``if __name__ ==
+ '__main__'`` line of the main module. For example::
+
+ from multiprocessing import Process, freeze_support
+
+ def f():
+ print 'hello world!'
+
+ if __name__ == '__main__':
+ freeze_support()
+ Process(target=f).start()
+
+ If the ``freeze_support()`` line is missed out then trying to run the frozen
+ executable will raise :exc:`RuntimeError`.
+
+ If the module is being run normally by the Python interpreter then
+ :func:`freeze_support` has no effect.
+
+.. function:: set_executable()
+
+ Sets the path of the python interpreter to use when starting a child process.
+ (By default :data:`sys.executable` is used). Embedders will probably need to
+ do some thing like ::
+
+ setExecutable(os.path.join(sys.exec_prefix, 'pythonw.exe'))
+
+ before they can create child processes. (Windows only)
+
+
+.. note::
+
+ :mod:`multiprocessing` contains no analogues of
+ :func:`threading.active_count`, :func:`threading.enumerate`,
+ :func:`threading.settrace`, :func:`threading.setprofile`,
+ :class:`threading.Timer`, or :class:`threading.local`.
+
+
+Connection Objects
+~~~~~~~~~~~~~~~~~~
+
+Connection objects allow the sending and receiving of picklable objects or
+strings. They can be thought of as message oriented connected sockets.
+
+Connection objects usually created using :func:`Pipe` -- see also
+:ref:`multiprocessing-listeners-clients`.
+
+.. class:: Connection
+
+ .. method:: send(obj)
+
+ Send an object to the other end of the connection which should be read
+ using :meth:`recv`.
+
+ The object must be picklable.
+
+ .. method:: recv()
+
+ Return an object sent from the other end of the connection using
+ :meth:`send`. Raises :exc:`EOFError` if there is nothing left to receive
+ and the other end was closed.
+
+ .. method:: fileno()
+
+ Returns the file descriptor or handle used by the connection.
+
+ .. method:: close()
+
+ Close the connection.
+
+ This is called automatically when the connection is garbage collected.
+
+ .. method:: poll([timeout])
+
+ Return whether there is any data available to be read.
+
+ If *timeout* is not specified then it will return immediately. If
+ *timeout* is a number then this specifies the maximum time in seconds to
+ block. If *timeout* is ``None`` then an infinite timeout is used.
+
+ .. method:: send_bytes(buffer[, offset[, size]])
+
+ Send byte data from an object supporting the buffer interface as a
+ complete message.
+
+ If *offset* is given then data is read from that position in *buffer*. If
+ *size* is given then that many bytes will be read from buffer.
+
+ .. method:: recv_bytes([maxlength])
+
+ Return a complete message of byte data sent from the other end of the
+ connection as a string. Raises :exc:`EOFError` if there is nothing left
+ to receive and the other end has closed.
+
+ If *maxlength* is specified and the message is longer than *maxlength*
+ then :exc:`IOError` is raised and the connection will no longer be
+ readable.
+
+ .. method:: recv_bytes_into(buffer[, offset])
+
+ Read into *buffer* a complete message of byte data sent from the other end
+ of the connection and return the number of bytes in the message. Raises
+ :exc:`EOFError` if there is nothing left to receive and the other end was
+ closed.
+
+ *buffer* must be an object satisfying the writable buffer interface. If
+ *offset* is given then the message will be written into the buffer from
+ *that position. Offset must be a non-negative integer less than the
+ *length of *buffer* (in bytes).
+
+ If the buffer is too short then a :exc:`BufferTooShort` exception is
+ raised and the complete message is available as ``e.args[0]`` where ``e``
+ is the exception instance.
+
+
+For example:
+
+ >>> from multiprocessing import Pipe
+ >>> a, b = Pipe()
+ >>> a.send([1, 'hello', None])
+ >>> b.recv()
+ [1, 'hello', None]
+ >>> b.send_bytes('thank you')
+ >>> a.recv_bytes()
+ 'thank you'
+ >>> import array
+ >>> arr1 = array.array('i', range(5))
+ >>> arr2 = array.array('i', [0] * 10)
+ >>> a.send_bytes(arr1)
+ >>> count = b.recv_bytes_into(arr2)
+ >>> assert count == len(arr1) * arr1.itemsize
+ >>> arr2
+ array('i', [0, 1, 2, 3, 4, 0, 0, 0, 0, 0])
+
+
+.. warning::
+
+ The :meth:`Connection.recv` method automatically unpickles the data it
+ receives, which can be a security risk unless you can trust the process
+ which sent the message.
+
+ Therefore, unless the connection object was produced using :func:`Pipe` you
+ should only use the :meth:`~Connection.recv` and :meth:`~Connection.send`
+ methods after performing some sort of authentication. See
+ :ref:`multiprocessing-auth-keys`.
+
+.. warning::
+
+ If a process is killed while it is trying to read or write to a pipe then
+ the data in the pipe is likely to become corrupted, because it may become
+ impossible to be sure where the message boundaries lie.
+
+
+Synchronization primitives
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Generally synchronization primitives are not as necessary in a multiprocess
+program as they are in a multithreaded program. See the documentation for
+:mod:`threading` module.
+
+Note that one can also create synchronization primitives by using a manager
+object -- see :ref:`multiprocessing-managers`.
+
+.. class:: BoundedSemaphore([value])
+
+ A bounded semaphore object: a clone of :class:`threading.BoundedSemaphore`.
+
+ (On Mac OS X this is indistinguishable from :class:`Semaphore` because
+ ``sem_getvalue()`` is not implemented on that platform).
+
+.. class:: Condition([lock])
+
+ A condition variable: a clone of :class:`threading.Condition`.
+
+ If *lock* is specified then it should be a :class:`Lock` or :class:`RLock`
+ object from :mod:`multiprocessing`.
+
+.. class:: Event()
+
+ A clone of :class:`threading.Event`.
+
+.. class:: Lock()
+
+ A non-recursive lock object: a clone of :class:`threading.Lock`.
+
+.. class:: RLock()
+
+ A recursive lock object: a clone of :class:`threading.RLock`.
+
+.. class:: Semaphore([value])
+
+ A bounded semaphore object: a clone of :class:`threading.Semaphore`.
+
+.. note::
+
+ The :meth:`acquire` method of :class:`BoundedSemaphore`, :class:`Lock`,
+ :class:`RLock` and :class:`Semaphore` has a timeout parameter not supported
+ by the equivalents in :mod:`threading`. The signature is
+ ``acquire(block=True, timeout=None)`` with keyword parameters being
+ acceptable. If *block* is ``True`` and *timeout* is not ``None`` then it
+ specifies a timeout in seconds. If *block* is ``False`` then *timeout* is
+ ignored.
+
+ Note that on OS/X ``sem_timedwait`` is unsupported, so timeout arguments
+ for these will be ignored.
+
+.. note::
+
+ If the SIGINT signal generated by Ctrl-C arrives while the main thread is
+ blocked by a call to :meth:`BoundedSemaphore.acquire`, :meth:`Lock.acquire`,
+ :meth:`RLock.acquire`, :meth:`Semaphore.acquire`, :meth:`Condition.acquire`
+ or :meth:`Condition.wait` then the call will be immediately interrupted and
+ :exc:`KeyboardInterrupt` will be raised.
+
+ This differs from the behaviour of :mod:`threading` where SIGINT will be
+ ignored while the equivalent blocking calls are in progress.
+
+
+Shared :mod:`ctypes` Objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is possible to create shared objects using shared memory which can be
+inherited by child processes.
+
+.. function:: Value(typecode_or_type[, *args, lock]])
+
+ Return a :mod:`ctypes` object allocated from shared memory. By default the
+ return value is actually a synchronized wrapper for the object.
+
+ *typecode_or_type* determines the type of the returned object: it is either a
+ ctypes type or a one character typecode of the kind used by the :mod:`array`
+ module. *\*args* is passed on to the constructor for the type.
+
+ If *lock* is ``True`` (the default) then a new lock object is created to
+ synchronize access to the value. If *lock* is a :class:`Lock` or
+ :class:`RLock` object then that will be used to synchronize access to the
+ value. If *lock* is ``False`` then access to the returned object will not be
+ automatically protected by a lock, so it will not necessarily be
+ "process-safe".
+
+ Note that *lock* is a keyword-only argument.
+
+.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True)
+
+ Return a ctypes array allocated from shared memory. By default the return
+ value is actually a synchronized wrapper for the array.
+
+ *typecode_or_type* determines the type of the elements of the returned array:
+ it is either a ctypes type or a one character typecode of the kind used by
+ the :mod:`array` module. If *size_or_initializer* is an integer, then it
+ determines the length of the array, and the array will be initially zeroed.
+ Otherwise, *size_or_initializer* is a sequence which is used to initialize
+ the array and whose length determines the length of the array.
+
+ If *lock* is ``True`` (the default) then a new lock object is created to
+ synchronize access to the value. If *lock* is a :class:`Lock` or
+ :class:`RLock` object then that will be used to synchronize access to the
+ value. If *lock* is ``False`` then access to the returned object will not be
+ automatically protected by a lock, so it will not necessarily be
+ "process-safe".
+
+ Note that *lock* is a keyword only argument.
+
+ Note that an array of :data:`ctypes.c_char` has *value* and *rawvalue*
+ attributes which allow one to use it to store and retrieve strings.
+
+
+The :mod:`multiprocessing.sharedctypes` module
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+.. module:: multiprocessing.sharedctypes
+ :synopsis: Allocate ctypes objects from shared memory.
+
+The :mod:`multiprocessing.sharedctypes` module provides functions for allocating
+:mod:`ctypes` objects from shared memory which can be inherited by child
+processes.
+
+.. note::
+
+ Although it is possible to store a pointer in shared memory remember that
+ this will refer to a location in the address space of a specific process.
+ However, the pointer is quite likely to be invalid in the context of a second
+ process and trying to dereference the pointer from the second process may
+ cause a crash.
+
+.. function:: RawArray(typecode_or_type, size_or_initializer)
+
+ Return a ctypes array allocated from shared memory.
+
+ *typecode_or_type* determines the type of the elements of the returned array:
+ it is either a ctypes type or a one character typecode of the kind used by
+ the :mod:`array` module. If *size_or_initializer* is an integer then it
+ determines the length of the array, and the array will be initially zeroed.
+ Otherwise *size_or_initializer* is a sequence which is used to initialize the
+ array and whose length determines the length of the array.
+
+ Note that setting and getting an element is potentially non-atomic -- use
+ :func:`Array` instead to make sure that access is automatically synchronized
+ using a lock.
+
+.. function:: RawValue(typecode_or_type, *args)
+
+ Return a ctypes object allocated from shared memory.
+
+ *typecode_or_type* determines the type of the returned object: it is either a
+ ctypes type or a one character typecode of the kind used by the :mod:`array`
+ module. */*args* is passed on to the constructor for the type.
+
+ Note that setting and getting the value is potentially non-atomic -- use
+ :func:`Value` instead to make sure that access is automatically synchronized
+ using a lock.
+
+ Note that an array of :data:`ctypes.c_char` has ``value`` and ``rawvalue``
+ attributes which allow one to use it to store and retrieve strings -- see
+ documentation for :mod:`ctypes`.
+
+.. function:: Array(typecode_or_type, size_or_initializer[, *args[, lock]])
+
+ The same as :func:`RawArray` except that depending on the value of *lock* a
+ process-safe synchronization wrapper may be returned instead of a raw ctypes
+ array.
+
+ If *lock* is ``True`` (the default) then a new lock object is created to
+ synchronize access to the value. If *lock* is a :class:`Lock` or
+ :class:`RLock` object then that will be used to synchronize access to the
+ value. If *lock* is ``False`` then access to the returned object will not be
+ automatically protected by a lock, so it will not necessarily be
+ "process-safe".
+
+ Note that *lock* is a keyword-only argument.
+
+.. function:: Value(typecode_or_type, *args[, lock])
+
+ The same as :func:`RawValue` except that depending on the value of *lock* a
+ process-safe synchronization wrapper may be returned instead of a raw ctypes
+ object.
+
+ If *lock* is ``True`` (the default) then a new lock object is created to
+ synchronize access to the value. If *lock* is a :class:`Lock` or
+ :class:`RLock` object then that will be used to synchronize access to the
+ value. If *lock* is ``False`` then access to the returned object will not be
+ automatically protected by a lock, so it will not necessarily be
+ "process-safe".
+
+ Note that *lock* is a keyword-only argument.
+
+.. function:: copy(obj)
+
+ Return a ctypes object allocated from shared memory which is a copy of the
+ ctypes object *obj*.
+
+.. function:: synchronized(obj[, lock])
+
+ Return a process-safe wrapper object for a ctypes object which uses *lock* to
+ synchronize access. If *lock* is ``None`` (the default) then a
+ :class:`multiprocessing.RLock` object is created automatically.
+
+ A synchronized wrapper will have two methods in addition to those of the
+ object it wraps: :meth:`get_obj` returns the wrapped object and
+ :meth:`get_lock` returns the lock object used for synchronization.
+
+ Note that accessing the ctypes object through the wrapper can be a lot slower
+ than accessing the raw ctypes object.
+
+
+The table below compares the syntax for creating shared ctypes objects from
+shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some
+subclass of :class:`ctypes.Structure`.)
+
+==================== ========================== ===========================
+ctypes sharedctypes using type sharedctypes using typecode
+==================== ========================== ===========================
+c_double(2.4) RawValue(c_double, 2.4) RawValue('d', 2.4)
+MyStruct(4, 6) RawValue(MyStruct, 4, 6)
+(c_short * 7)() RawArray(c_short, 7) RawArray('h', 7)
+(c_int * 3)(9, 2, 8) RawArray(c_int, (9, 2, 8)) RawArray('i', (9, 2, 8))
+==================== ========================== ===========================
+
+
+Below is an example where a number of ctypes objects are modified by a child
+process::
+
+ from multiprocessing import Process, Lock
+ from multiprocessing.sharedctypes import Value, Array
+ from ctypes import Structure, c_double
+
+ class Point(Structure):
+ _fields_ = [('x', c_double), ('y', c_double)]
+
+ def modify(n, x, s, A):
+ n.value **= 2
+ x.value **= 2
+ s.value = s.value.upper()
+ for a in A:
+ a.x **= 2
+ a.y **= 2
+
+ if __name__ == '__main__':
+ lock = Lock()
+
+ n = Value('i', 7)
+ x = Value(ctypes.c_double, 1.0/3.0, lock=False)
+ s = Array('c', 'hello world', lock=lock)
+ A = Array(Point, [(1.875,-6.25), (-5.75,2.0), (2.375,9.5)], lock=lock)
+
+ p = Process(target=modify, args=(n, x, s, A))
+ p.start()
+ p.join()
+
+ print n.value
+ print x.value
+ print s.value
+ print [(a.x, a.y) for a in A]
+
+
+.. highlightlang:: none
+
+The results printed are ::
+
+ 49
+ 0.1111111111111111
+ HELLO WORLD
+ [(3.515625, 39.0625), (33.0625, 4.0), (5.640625, 90.25)]
+
+.. highlightlang:: python
+
+
+.. _multiprocessing-managers:
+
+Managers
+~~~~~~~~
+
+Managers provide a way to create data which can be shared between different
+processes. A manager object controls a server process which manages *shared
+objects*. Other processes can access the shared objects by using proxies.
+
+.. function:: multiprocessing.Manager()
+
+ Returns a started :class:`~multiprocessing.managers.SyncManager` object which
+ can be used for sharing objects between processes. The returned manager
+ object corresponds to a spawned child process and has methods which will
+ create shared objects and return corresponding proxies.
+
+.. module:: multiprocessing.managers
+ :synopsis: Share data between process with shared objects.
+
+Manager processes will be shutdown as soon as they are garbage collected or
+their parent process exits. The manager classes are defined in the
+:mod:`multiprocessing.managers` module:
+
+.. class:: BaseManager([address[, authkey]])
+
+ Create a BaseManager object.
+
+ Once created one should call :meth:`start` or :meth:`serve_forever` to ensure
+ that the manager object refers to a started manager process.
+
+ *address* is the address on which the manager process listens for new
+ connections. If *address* is ``None`` then an arbitrary one is chosen.
+
+ *authkey* is the authentication key which will be used to check the validity
+ of incoming connections to the server process. If *authkey* is ``None`` then
+ ``current_process().authkey``. Otherwise *authkey* is used and it
+ must be a string.
+
+ .. method:: start()
+
+ Start a subprocess to start the manager.
+
+ .. method:: serve_forever()
+
+ Run the server in the current process.
+
+ .. method:: from_address(address, authkey)
+
+ A class method which creates a manager object referring to a pre-existing
+ server process which is using the given address and authentication key.
+
+ .. method:: get_server()
+
+ Returns a :class:`Server` object which represents the actual server under
+ the control of the Manager. The :class:`Server` object supports the
+ :meth:`serve_forever` method::
+
+ >>> from multiprocessing.managers import BaseManager
+ >>> m = BaseManager(address=('', 50000), authkey='abc'))
+ >>> server = m.get_server()
+ >>> s.serve_forever()
+
+ :class:`Server` additionally have an :attr:`address` attribute.
+
+ .. method:: connect()
+
+ Connect a local manager object to a remote manager process::
+
+ >>> from multiprocessing.managers import BaseManager
+ >>> m = BaseManager(address='127.0.0.1', authkey='abc))
+ >>> m.connect()
+
+ .. method:: shutdown()
+
+ Stop the process used by the manager. This is only available if
+ :meth:`start` has been used to start the server process.
+
+ This can be called multiple times.
+
+ .. method:: register(typeid[, callable[, proxytype[, exposed[, method_to_typeid[, create_method]]]]])
+
+ A classmethod which can be used for registering a type or callable with
+ the manager class.
+
+ *typeid* is a "type identifier" which is used to identify a particular
+ type of shared object. This must be a string.
+
+ *callable* is a callable used for creating objects for this type
+ identifier. If a manager instance will be created using the
+ :meth:`from_address` classmethod or if the *create_method* argument is
+ ``False`` then this can be left as ``None``.
+
+ *proxytype* is a subclass of :class:`BaseProxy` which is used to create
+ proxies for shared objects with this *typeid*. If ``None`` then a proxy
+ class is created automatically.
+
+ *exposed* is used to specify a sequence of method names which proxies for
+ this typeid should be allowed to access using
+ :meth:`BaseProxy._callMethod`. (If *exposed* is ``None`` then
+ :attr:`proxytype._exposed_` is used instead if it exists.) In the case
+ where no exposed list is specified, all "public methods" of the shared
+ object will be accessible. (Here a "public method" means any attribute
+ which has a :meth:`__call__` method and whose name does not begin with
+ ``'_'``.)
+
+ *method_to_typeid* is a mapping used to specify the return type of those
+ exposed methods which should return a proxy. It maps method names to
+ typeid strings. (If *method_to_typeid* is ``None`` then
+ :attr:`proxytype._method_to_typeid_` is used instead if it exists.) If a
+ method's name is not a key of this mapping or if the mapping is ``None``
+ then the object returned by the method will be copied by value.
+
+ *create_method* determines whether a method should be created with name
+ *typeid* which can be used to tell the server process to create a new
+ shared object and return a proxy for it. By default it is ``True``.
+
+ :class:`BaseManager` instances also have one read-only property:
+
+ .. attribute:: address
+
+ The address used by the manager.
+
+
+.. class:: SyncManager
+
+ A subclass of :class:`BaseManager` which can be used for the synchronization
+ of processes. Objects of this type are returned by
+ :func:`multiprocessing.Manager`.
+
+ It also supports creation of shared lists and dictionaries.
+
+ .. method:: BoundedSemaphore([value])
+
+ Create a shared :class:`threading.BoundedSemaphore` object and return a
+ proxy for it.
+
+ .. method:: Condition([lock])
+
+ Create a shared :class:`threading.Condition` object and return a proxy for
+ it.
+
+ If *lock* is supplied then it should be a proxy for a
+ :class:`threading.Lock` or :class:`threading.RLock` object.
+
+ .. method:: Event()
+
+ Create a shared :class:`threading.Event` object and return a proxy for it.
+
+ .. method:: Lock()
+
+ Create a shared :class:`threading.Lock` object and return a proxy for it.
+
+ .. method:: Namespace()
+
+ Create a shared :class:`Namespace` object and return a proxy for it.
+
+ .. method:: Queue([maxsize])
+
+ Create a shared :class:`Queue.Queue` object and return a proxy for it.
+
+ .. method:: RLock()
+
+ Create a shared :class:`threading.RLock` object and return a proxy for it.
+
+ .. method:: Semaphore([value])
+
+ Create a shared :class:`threading.Semaphore` object and return a proxy for
+ it.
+
+ .. method:: Array(typecode, sequence)
+
+ Create an array and return a proxy for it.
+
+ .. method:: Value(typecode, value)
+
+ Create an object with a writable ``value`` attribute and return a proxy
+ for it.
+
+ .. method:: dict()
+ dict(mapping)
+ dict(sequence)
+
+ Create a shared ``dict`` object and return a proxy for it.
+
+ .. method:: list()
+ list(sequence)
+
+ Create a shared ``list`` object and return a proxy for it.
+
+
+Namespace objects
+>>>>>>>>>>>>>>>>>
+
+A namespace object has no public methods, but does have writable attributes.
+Its representation shows the values of its attributes.
+
+However, when using a proxy for a namespace object, an attribute beginning with
+``'_'`` will be an attribute of the proxy and not an attribute of the referent::
+
+ >>> manager = multiprocessing.Manager()
+ >>> Global = manager.Namespace()
+ >>> Global.x = 10
+ >>> Global.y = 'hello'
+ >>> Global._z = 12.3 # this is an attribute of the proxy
+ >>> print Global
+ Namespace(x=10, y='hello')
+
+
+Customized managers
+>>>>>>>>>>>>>>>>>>>
+
+To create one's own manager, one creates a subclass of :class:`BaseManager` and
+use the :meth:`~BaseManager.resgister` classmethod to register new types or
+callables with the manager class. For example::
+
+ from multiprocessing.managers import BaseManager
+
+ class MathsClass(object):
+ def add(self, x, y):
+ return x + y
+ def mul(self, x, y):
+ return x * y
+
+ class MyManager(BaseManager):
+ pass
+
+ MyManager.register('Maths', MathsClass)
+
+ if __name__ == '__main__':
+ manager = MyManager()
+ manager.start()
+ maths = manager.Maths()
+ print maths.add(4, 3) # prints 7
+ print maths.mul(7, 8) # prints 56
+
+
+Using a remote manager
+>>>>>>>>>>>>>>>>>>>>>>
+
+It is possible to run a manager server on one machine and have clients use it
+from other machines (assuming that the firewalls involved allow it).
+
+Running the following commands creates a server for a single shared queue which
+remote clients can access::
+
+ >>> from multiprocessing.managers import BaseManager
+ >>> import Queue
+ >>> queue = Queue.Queue()
+ >>> class QueueManager(BaseManager): pass
+ ...
+ >>> QueueManager.register('get_queue', callable=lambda:queue)
+ >>> m = QueueManager(address=('', 50000), authkey='abracadabra')
+ >>> s = m.get_server()
+ >>> s.serveForever()
+
+One client can access the server as follows::
+
+ >>> from multiprocessing.managers import BaseManager
+ >>> class QueueManager(BaseManager): pass
+ ...
+ >>> QueueManager.register('get_queue')
+ >>> m = QueueManager(address=('foo.bar.org', 50000), authkey='abracadabra')
+ >>> m.connect()
+ >>> queue = m.get_queue()
+ >>> queue.put('hello')
+
+Another client can also use it::
+
+ >>> from multiprocessing.managers import BaseManager
+ >>> class QueueManager(BaseManager): pass
+ ...
+ >>> QueueManager.register('getQueue')
+ >>> m = QueueManager.from_address(address=('foo.bar.org', 50000), authkey='abracadabra')
+ >>> queue = m.getQueue()
+ >>> queue.get()
+ 'hello'
+
+Local processes can also access that queue, using the code from above on the
+client to access it remotely::
+
+ >>> from multiprocessing import Process, Queue
+ >>> from multiprocessing.managers import BaseManager
+ >>> class Worker(Process):
+ ... def __init__(self, q):
+ ... self.q = q
+ ... super(Worker, self).__init__()
+ ... def run(self):
+ ... self.q.put('local hello')
+ ...
+ >>> queue = Queue()
+ >>> w = Worker(queue)
+ >>> w.start()
+ >>> class QueueManager(BaseManager): pass
+ ...
+ >>> QueueManager.register('get_queue', callable=lambda: queue)
+ >>> m = QueueManager(address=('', 50000), authkey='abracadabra')
+ >>> s = m.get_server()
+ >>> s.serve_forever()
+
+Proxy Objects
+~~~~~~~~~~~~~
+
+A proxy is an object which *refers* to a shared object which lives (presumably)
+in a different process. The shared object is said to be the *referent* of the
+proxy. Multiple proxy objects may have the same referent.
+
+A proxy object has methods which invoke corresponding methods of its referent
+(although not every method of the referent will necessarily be available through
+the proxy). A proxy can usually be used in most of the same ways that its
+referent can::
+
+ >>> from multiprocessing import Manager
+ >>> manager = Manager()
+ >>> l = manager.list([i*i for i in range(10)])
+ >>> print l
+ [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
+ >>> print repr(l)
+ <ListProxy object, typeid 'list' at 0xb799974c>
+ >>> l[4]
+ 16
+ >>> l[2:5]
+ [4, 9, 16]
+
+Notice that applying :func:`str` to a proxy will return the representation of
+the referent, whereas applying :func:`repr` will return the representation of
+the proxy.
+
+An important feature of proxy objects is that they are picklable so they can be
+passed between processes. Note, however, that if a proxy is sent to the
+corresponding manager's process then unpickling it will produce the referent
+itself. This means, for example, that one shared object can contain a second::
+
+ >>> a = manager.list()
+ >>> b = manager.list()
+ >>> a.append(b) # referent of a now contains referent of b
+ >>> print a, b
+ [[]] []
+ >>> b.append('hello')
+ >>> print a, b
+ [['hello']] ['hello']
+
+.. note::
+
+ The proxy types in :mod:`multiprocessing` do nothing to support comparisons
+ by value. So, for instance, ::
+
+ manager.list([1,2,3]) == [1,2,3]
+
+ will return ``False``. One should just use a copy of the referent instead
+ when making comparisons.
+
+.. class:: BaseProxy
+
+ Proxy objects are instances of subclasses of :class:`BaseProxy`.
+
+ .. method:: _call_method(methodname[, args[, kwds]])
+
+ Call and return the result of a method of the proxy's referent.
+
+ If ``proxy`` is a proxy whose referent is ``obj`` then the expression ::
+
+ proxy._call_method(methodname, args, kwds)
+
+ will evaluate the expression ::
+
+ getattr(obj, methodname)(*args, **kwds)
+
+ in the manager's process.
+
+ The returned value will be a copy of the result of the call or a proxy to
+ a new shared object -- see documentation for the *method_to_typeid*
+ argument of :meth:`BaseManager.register`.
+
+ If an exception is raised by the call, then then is re-raised by
+ :meth:`_call_method`. If some other exception is raised in the manager's
+ process then this is converted into a :exc:`RemoteError` exception and is
+ raised by :meth:`_call_method`.
+
+ Note in particular that an exception will be raised if *methodname* has
+ not been *exposed*
+
+ An example of the usage of :meth:`_call_method`::
+
+ >>> l = manager.list(range(10))
+ >>> l._call_method('__len__')
+ 10
+ >>> l._call_method('__getslice__', (2, 7)) # equiv to `l[2:7]`
+ [2, 3, 4, 5, 6]
+ >>> l._call_method('__getitem__', (20,)) # equiv to `l[20]`
+ Traceback (most recent call last):
+ ...
+ IndexError: list index out of range
+
+ .. method:: _get_value()
+
+ Return a copy of the referent.
+
+ If the referent is unpicklable then this will raise an exception.
+
+ .. method:: __repr__
+
+ Return a representation of the proxy object.
+
+ .. method:: __str__
+
+ Return the representation of the referent.
+
+
+Cleanup
+>>>>>>>
+
+A proxy object uses a weakref callback so that when it gets garbage collected it
+deregisters itself from the manager which owns its referent.
+
+A shared object gets deleted from the manager process when there are no longer
+any proxies referring to it.
+
+
+Process Pools
+~~~~~~~~~~~~~
+
+.. module:: multiprocessing.pool
+ :synopsis: Create pools of processes.
+
+One can create a pool of processes which will carry out tasks submitted to it
+with the :class:`Pool` class.
+
+.. class:: multiprocessing.Pool([processes[, initializer[, initargs]]])
+
+ A process pool object which controls a pool of worker processes to which jobs
+ can be submitted. It supports asynchronous results with timeouts and
+ callbacks and has a parallel map implementation.
+
+ *processes* is the number of worker processes to use. If *processes* is
+ ``None`` then the number returned by :func:`cpu_count` is used. If
+ *initializer* is not ``None`` then each worker process will call
+ ``initializer(*initargs)`` when it starts.
+
+ .. method:: apply(func[, args[, kwds]])
+
+ Equivalent of the :func:`apply` builtin function. It blocks till the
+ result is ready.
+
+ .. method:: apply_async(func[, args[, kwds[, callback]]])
+
+ A variant of the :meth:`apply` method which returns a result object.
+
+ If *callback* is specified then it should be a callable which accepts a
+ single argument. When the result becomes ready *callback* is applied to
+ it (unless the call failed). *callback* should complete immediately since
+ otherwise the thread which handles the results will get blocked.
+
+ .. method:: map(func, iterable[, chunksize])
+
+ A parallel equivalent of the :func:`map` builtin function. It blocks till
+ the result is ready.
+
+ This method chops the iterable into a number of chunks which it submits to
+ the process pool as separate tasks. The (approximate) size of these
+ chunks can be specified by setting *chunksize* to a positive integer.
+
+ .. method:: map_async(func, iterable[, chunksize[, callback]])
+
+ A variant of the :meth:`map` method which returns a result object.
+
+ If *callback* is specified then it should be a callable which accepts a
+ single argument. When the result becomes ready *callback* is applied to
+ it (unless the call failed). *callback* should complete immediately since
+ otherwise the thread which handles the results will get blocked.
+
+ .. method:: imap(func, iterable[, chunksize])
+
+ An equivalent of :func:`itertools.imap`.
+
+ The *chunksize* argument is the same as the one used by the :meth:`.map`
+ method. For very long iterables using a large value for *chunksize* can
+ make make the job complete **much** faster than using the default value of
+ ``1``.
+
+ Also if *chunksize* is ``1`` then the :meth:`next` method of the iterator
+ returned by the :meth:`imap` method has an optional *timeout* parameter:
+ ``next(timeout)`` will raise :exc:`multiprocessing.TimeoutError` if the
+ result cannot be returned within *timeout* seconds.
+
+ .. method:: imap_unordered(func, iterable[, chunksize])
+
+ The same as :meth:`imap` except that the ordering of the results from the
+ returned iterator should be considered arbitrary. (Only when there is
+ only one worker process is the order guaranteed to be "correct".)
+
+ .. method:: close()
+
+ Prevents any more tasks from being submitted to the pool. Once all the
+ tasks have been completed the worker processes will exit.
+
+ .. method:: terminate()
+
+ Stops the worker processes immediately without completing outstanding
+ work. When the pool object is garbage collected :meth:`terminate` will be
+ called immediately.
+
+ .. method:: join()
+
+ Wait for the worker processes to exit. One must call :meth:`close` or
+ :meth:`terminate` before using :meth:`join`.
+
+
+.. class:: AsyncResult
+
+ The class of the result returned by :meth:`Pool.apply_async` and
+ :meth:`Pool.map_async`.
+
+ .. method:: get([timeout])
+
+ Return the result when it arrives. If *timeout* is not ``None`` and the
+ result does not arrive within *timeout* seconds then
+ :exc:`multiprocessing.TimeoutError` is raised. If the remote call raised
+ an exception then that exception will be reraised by :meth:`get`.
+
+ .. method:: wait([timeout])
+
+ Wait until the result is available or until *timeout* seconds pass.
+
+ .. method:: ready()
+
+ Return whether the call has completed.
+
+ .. method:: successful()
+
+ Return whether the call completed without raising an exception. Will
+ raise :exc:`AssertionError` if the result is not ready.
+
+The following example demonstrates the use of a pool::
+
+ from multiprocessing import Pool
+
+ def f(x):
+ return x*x
+
+ if __name__ == '__main__':
+ pool = Pool(processes=4) # start 4 worker processes
+
+ result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously
+ print result.get(timeout=1) # prints "100" unless your computer is *very* slow
+
+ print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
+
+ it = pool.imap(f, range(10))
+ print it.next() # prints "0"
+ print it.next() # prints "1"
+ print it.next(timeout=1) # prints "4" unless your computer is *very* slow
+
+ import time
+ result = pool.apply_async(time.sleep, (10,))
+ print result.get(timeout=1) # raises TimeoutError
+
+
+.. _multiprocessing-listeners-clients:
+
+Listeners and Clients
+~~~~~~~~~~~~~~~~~~~~~
+
+.. module:: multiprocessing.connection
+ :synopsis: API for dealing with sockets.
+
+Usually message passing between processes is done using queues or by using
+:class:`Connection` objects returned by :func:`Pipe`.
+
+However, the :mod:`multiprocessing.connection` module allows some extra
+flexibility. It basically gives a high level message oriented API for dealing
+with sockets or Windows named pipes, and also has support for *digest
+authentication* using the :mod:`hmac` module.
+
+
+.. function:: deliver_challenge(connection, authkey)
+
+ Send a randomly generated message to the other end of the connection and wait
+ for a reply.
+
+ If the reply matches the digest of the message using *authkey* as the key
+ then a welcome message is sent to the other end of the connection. Otherwise
+ :exc:`AuthenticationError` is raised.
+
+.. function:: answerChallenge(connection, authkey)
+
+ Receive a message, calculate the digest of the message using *authkey* as the
+ key, and then send the digest back.
+
+ If a welcome message is not received, then :exc:`AuthenticationError` is
+ raised.
+
+.. function:: Client(address[, family[, authenticate[, authkey]]])
+
+ Attempt to set up a connection to the listener which is using address
+ *address*, returning a :class:`~multiprocessing.Connection`.
+
+ The type of the connection is determined by *family* argument, but this can
+ generally be omitted since it can usually be inferred from the format of
+ *address*. (See :ref:`multiprocessing-address-formats`)
+
+ If *authentication* is ``True`` or *authkey* is a string then digest
+ authentication is used. The key used for authentication will be either
+ *authkey* or ``current_process().authkey)`` if *authkey* is ``None``.
+ If authentication fails then :exc:`AuthenticationError` is raised. See
+ :ref:`multiprocessing-auth-keys`.
+
+.. class:: Listener([address[, family[, backlog[, authenticate[, authkey]]]]])
+
+ A wrapper for a bound socket or Windows named pipe which is 'listening' for
+ connections.
+
+ *address* is the address to be used by the bound socket or named pipe of the
+ listener object.
+
+ *family* is the type of socket (or named pipe) to use. This can be one of
+ the strings ``'AF_INET'`` (for a TCP socket), ``'AF_UNIX'`` (for a Unix
+ domain socket) or ``'AF_PIPE'`` (for a Windows named pipe). Of these only
+ the first is guaranteed to be available. If *family* is ``None`` then the
+ family is inferred from the format of *address*. If *address* is also
+ ``None`` then a default is chosen. This default is the family which is
+ assumed to be the fastest available. See
+ :ref:`multiprocessing-address-formats`. Note that if *family* is
+ ``'AF_UNIX'`` and address is ``None`` then the socket will be created in a
+ private temporary directory created using :func:`tempfile.mkstemp`.
+
+ If the listener object uses a socket then *backlog* (1 by default) is passed
+ to the :meth:`listen` method of the socket once it has been bound.
+
+ If *authenticate* is ``True`` (``False`` by default) or *authkey* is not
+ ``None`` then digest authentication is used.
+
+ If *authkey* is a string then it will be used as the authentication key;
+ otherwise it must be *None*.
+
+ If *authkey* is ``None`` and *authenticate* is ``True`` then
+ ``current_process().authkey`` is used as the authentication key. If
+ *authkey* is ``None`` and *authentication* is ``False`` then no
+ authentication is done. If authentication fails then
+ :exc:`AuthenticationError` is raised. See :ref:`multiprocessing-auth-keys`.
+
+ .. method:: accept()
+
+ Accept a connection on the bound socket or named pipe of the listener
+ object and return a :class:`Connection` object. If authentication is
+ attempted and fails, then :exc:`AuthenticationError` is raised.
+
+ .. method:: close()
+
+ Close the bound socket or named pipe of the listener object. This is
+ called automatically when the listener is garbage collected. However it
+ is advisable to call it explicitly.
+
+ Listener objects have the following read-only properties:
+
+ .. attribute:: address
+
+ The address which is being used by the Listener object.
+
+ .. attribute:: last_accepted
+
+ The address from which the last accepted connection came. If this is
+ unavailable then it is ``None``.
+
+
+The module defines two exceptions:
+
+.. exception:: AuthenticationError
+
+ Exception raised when there is an authentication error.
+
+
+**Examples**
+
+The following server code creates a listener which uses ``'secret password'`` as
+an authentication key. It then waits for a connection and sends some data to
+the client::
+
+ from multiprocessing.connection import Listener
+ from array import array
+
+ address = ('localhost', 6000) # family is deduced to be 'AF_INET'
+ listener = Listener(address, authkey='secret password')
+
+ conn = listener.accept()
+ print 'connection accepted from', listener.last_accepted
+
+ conn.send([2.25, None, 'junk', float])
+
+ conn.send_bytes('hello')
+
+ conn.send_bytes(array('i', [42, 1729]))
+
+ conn.close()
+ listener.close()
+
+The following code connects to the server and receives some data from the
+server::
+
+ from multiprocessing.connection import Client
+ from array import array
+
+ address = ('localhost', 6000)
+ conn = Client(address, authkey='secret password')
+
+ print conn.recv() # => [2.25, None, 'junk', float]
+
+ print conn.recv_bytes() # => 'hello'
+
+ arr = array('i', [0, 0, 0, 0, 0])
+ print conn.recv_bytes_into(arr) # => 8
+ print arr # => array('i', [42, 1729, 0, 0, 0])
+
+ conn.close()
+
+
+.. _multiprocessing-address-formats:
+
+Address Formats
+>>>>>>>>>>>>>>>
+
+* An ``'AF_INET'`` address is a tuple of the form ``(hostname, port)`` where
+ *hostname* is a string and *port* is an integer.
+
+* An ``'AF_UNIX'`` address is a string representing a filename on the
+ filesystem.
+
+* An ``'AF_PIPE'`` address is a string of the form
+ ``r'\\\\.\\pipe\\PipeName'``. To use :func:`Client` to connect to a named
+ pipe on a remote computer called ServerName* one should use an address of the
+ form ``r'\\\\ServerName\\pipe\\PipeName'`` instead.
+
+Note that any string beginning with two backslashes is assumed by default to be
+an ``'AF_PIPE'`` address rather than an ``'AF_UNIX'`` address.
+
+
+.. _multiprocessing-auth-keys:
+
+Authentication keys
+~~~~~~~~~~~~~~~~~~~
+
+When one uses :meth:`Connection.recv`, the data received is automatically
+unpickled. Unfortunately unpickling data from an untrusted source is a security
+risk. Therefore :class:`Listener` and :func:`Client` use the :mod:`hmac` module
+to provide digest authentication.
+
+An authentication key is a string which can be thought of as a password: once a
+connection is established both ends will demand proof that the other knows the
+authentication key. (Demonstrating that both ends are using the same key does
+**not** involve sending the key over the connection.)
+
+If authentication is requested but do authentication key is specified then the
+return value of ``current_process().authkey`` is used (see
+:class:`~multiprocessing.Process`). This value will automatically inherited by
+any :class:`~multiprocessing.Process` object that the current process creates.
+This means that (by default) all processes of a multi-process program will share
+a single authentication key which can be used when setting up connections
+between the themselves.
+
+Suitable authentication keys can also be generated by using :func:`os.urandom`.
+
+
+Logging
+~~~~~~~
+
+Some support for logging is available. Note, however, that the :mod:`logging`
+package does not use process shared locks so it is possible (depending on the
+handler type) for messages from different processes to get mixed up.
+
+.. currentmodule:: multiprocessing
+.. function:: get_logger()
+
+ Returns the logger used by :mod:`multiprocessing`. If necessary, a new one
+ will be created.
+
+ When first created the logger has level :data:`logging.NOTSET` and has a
+ handler which sends output to :data:`sys.stderr` using format
+ ``'[%(levelname)s/%(processName)s] %(message)s'``. (The logger allows use of
+ the non-standard ``'%(processName)s'`` format.) Message sent to this logger
+ will not by default propagate to the root logger.
+
+ Note that on Windows child processes will only inherit the level of the
+ parent process's logger -- any other customization of the logger will not be
+ inherited.
+
+Below is an example session with logging turned on::
+
+ >>> import multiprocessing, logging
+ >>> logger = multiprocessing.getLogger()
+ >>> logger.setLevel(logging.INFO)
+ >>> logger.warning('doomed')
+ [WARNING/MainProcess] doomed
+ >>> m = multiprocessing.Manager()
+ [INFO/SyncManager-1] child process calling self.run()
+ [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa'
+ >>> del m
+ [INFO/MainProcess] sending shutdown message to manager
+ [INFO/SyncManager-1] manager exiting with exitcode 0
+
+
+The :mod:`multiprocessing.dummy` module
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. module:: multiprocessing.dummy
+ :synopsis: Dumb wrapper around threading.
+
+:mod:`multiprocessing.dummy` replicates the API of :mod:`multiprocessing` but is
+no more than a wrapper around the :mod:`threading` module.
+
+
+.. _multiprocessing-programming:
+
+Programming guidelines
+----------------------
+
+There are certain guidelines and idioms which should be adhered to when using
+:mod:`multiprocessing`.
+
+
+All platforms
+~~~~~~~~~~~~~
+
+Avoid shared state
+
+ As far as possible one should try to avoid shifting large amounts of data
+ between processes.
+
+ It is probably best to stick to using queues or pipes for communication
+ between processes rather than using the lower level synchronization
+ primitives from the :mod:`threading` module.
+
+Picklability
+
+ Ensure that the arguments to the methods of proxies are picklable.
+
+Thread safety of proxies
+
+ Do not use a proxy object from more than one thread unless you protect it
+ with a lock.
+
+ (There is never a problem with different processes using the *same* proxy.)
+
+Joining zombie processes
+
+ On Unix when a process finishes but has not been joined it becomes a zombie.
+ There should never be very many because each time a new process starts (or
+ :func:`active_children` is called) all completed processes which have not
+ yet been joined will be joined. Also calling a finished process's
+ :meth:`Process.is_alive` will join the process. Even so it is probably good
+ practice to explicitly join all the processes that you start.
+
+Better to inherit than pickle/unpickle
+
+ On Windows many types from :mod:`multiprocessing` need to be picklable so
+ that child processes can use them. However, one should generally avoid
+ sending shared objects to other processes using pipes or queues. Instead
+ you should arrange the program so that a process which need access to a
+ shared resource created elsewhere can inherit it from an ancestor process.
+
+Avoid terminating processes
+
+ Using the :meth:`Process.terminate` method to stop a process is liable to
+ cause any shared resources (such as locks, semaphores, pipes and queues)
+ currently being used by the process to become broken or unavailable to other
+ processes.
+
+ Therefore it is probably best to only consider using
+ :meth:`Process.terminate` on processes which never use any shared resources.
+
+Joining processes that use queues
+
+ Bear in mind that a process that has put items in a queue will wait before
+ terminating until all the buffered items are fed by the "feeder" thread to
+ the underlying pipe. (The child process can call the
+ :meth:`Queue.cancel_join_thread` method of the queue to avoid this behaviour.)
+
+ This means that whenever you use a queue you need to make sure that all
+ items which have been put on the queue will eventually be removed before the
+ process is joined. Otherwise you cannot be sure that processes which have
+ put items on the queue will terminate. Remember also that non-daemonic
+ processes will be automatically be joined.
+
+ An example which will deadlock is the following::
+
+ from multiprocessing import Process, Queue
+
+ def f(q):
+ q.put('X' * 1000000)
+
+ if __name__ == '__main__':
+ queue = Queue()
+ p = Process(target=f, args=(queue,))
+ p.start()
+ p.join() # this deadlocks
+ obj = queue.get()
+
+ A fix here would be to swap the last two lines round (or simply remove the
+ ``p.join()`` line).
+
+Explicitly pass resources to child processes
+
+ On Unix a child process can make use of a shared resource created in a
+ parent process using a global resource. However, it is better to pass the
+ object as an argument to the constructor for the child process.
+
+ Apart from making the code (potentially) compatible with Windows this also
+ ensures that as long as the child process is still alive the object will not
+ be garbage collected in the parent process. This might be important if some
+ resource is freed when the object is garbage collected in the parent
+ process.
+
+ So for instance ::
+
+ from multiprocessing import Process, Lock
+
+ def f():
+ ... do something using "lock" ...
+
+ if __name__ == '__main__':
+ lock = Lock()
+ for i in range(10):
+ Process(target=f).start()
+
+ should be rewritten as ::
+
+ from multiprocessing import Process, Lock
+
+ def f(l):
+ ... do something using "l" ...
+
+ if __name__ == '__main__':
+ lock = Lock()
+ for i in range(10):
+ Process(target=f, args=(lock,)).start()
+
+
+Windows
+~~~~~~~
+
+Since Windows lacks :func:`os.fork` it has a few extra restrictions:
+
+More picklability
+
+ Ensure that all arguments to :meth:`Process.__init__` are picklable. This
+ means, in particular, that bound or unbound methods cannot be used directly
+ as the ``target`` argument on Windows --- just define a function and use
+ that instead.
+
+ Also, if you subclass :class:`Process` then make sure that instances will be
+ picklable when the :meth:`Process.start` method is called.
+
+Global variables
+
+ Bear in mind that if code run in a child process tries to access a global
+ variable, then the value it sees (if any) may not be the same as the value
+ in the parent process at the time that :meth:`Process.start` was called.
+
+ However, global variables which are just module level constants cause no
+ problems.
+
+Safe importing of main module
+
+ Make sure that the main module can be safely imported by a new Python
+ interpreter without causing unintended side effects (such a starting a new
+ process).
+
+ For example, under Windows running the following module would fail with a
+ :exc:`RuntimeError`::
+
+ from multiprocessing import Process
+
+ def foo():
+ print 'hello'
+
+ p = Process(target=foo)
+ p.start()
+
+ Instead one should protect the "entry point" of the program by using ``if
+ __name__ == '__main__':`` as follows::
+
+ from multiprocessing import Process, freeze_support
+
+ def foo():
+ print 'hello'
+
+ if __name__ == '__main__':
+ freeze_support()
+ p = Process(target=foo)
+ p.start()
+
+ (The ``freeze_support()`` line can be omitted if the program will be run
+ normally instead of frozen.)
+
+ This allows the newly spawned Python interpreter to safely import the module
+ and then run the module's ``foo()`` function.
+
+ Similar restrictions apply if a pool or manager is created in the main
+ module.
+
+
+.. _multiprocessing-examples:
+
+Examples
+--------
+
+Demonstration of how to create and use customized managers and proxies:
+
+.. literalinclude:: ../includes/mp_newtype.py
+
+
+Using :class:`Pool`:
+
+.. literalinclude:: ../includes/mp_pool.py
+
+
+Synchronization types like locks, conditions and queues:
+
+.. literalinclude:: ../includes/mp_synchronize.py
+
+
+An showing how to use queues to feed tasks to a collection of worker process and
+collect the results:
+
+.. literalinclude:: ../includes/mp_workers.py
+
+
+An example of how a pool of worker processes can each run a
+:class:`SimpleHTTPServer.HttpServer` instance while sharing a single listening
+socket.
+
+.. literalinclude:: ../includes/mp_webserver.py
+
+
+Some simple benchmarks comparing :mod:`multiprocessing` with :mod:`threading`:
+
+.. literalinclude:: ../includes/mp_benchmarks.py
+
+An example/demo of how to use the :class:`managers.SyncManager`, :class:`Process`
+and others to build a system which can distribute processes and work via a
+distributed queue to a "cluster" of machines on a network, accessible via SSH.
+You will need to have private key authentication for all hosts configured for
+this to work.
+
+.. literalinclude:: ../includes/mp_distributing.py