:mod:`mailbox` --- Manipulate mailboxes in various formats

==========================================================

.. module:: mailbox

   :synopsis: Manipulate mailboxes in various formats

.. moduleauthor:: Gregory K. Johnson <gkj@gregorykjohnson.com>

.. sectionauthor:: Gregory K. Johnson <gkj@gregorykjohnson.com>

This module defines two classes, :class:`Mailbox` and :class:`Message`, for

accessing and manipulating on-disk mailboxes and the messages they contain.

:class:`Mailbox` offers a dictionary-like mapping from keys to messages.

:class:`Message` extends the :mod:`email.Message` module's :class:`Message`

class with format-specific state and behavior. Supported mailbox formats are

Maildir, mbox, MH, Babyl, and MMDF.

.. seealso::

   Module :mod:`email`

      Represent and manipulate messages.

.. _mailbox-objects:

:class:`Mailbox` objects

------------------------

.. class:: Mailbox

   A mailbox, which may be inspected and modified.

   The :class:`Mailbox` class defines an interface and is not intended to be

   instantiated.  Instead, format-specific subclasses should inherit from

   :class:`Mailbox` and your code should instantiate a particular subclass.

   The :class:`Mailbox` interface is dictionary-like, with small keys

   corresponding to messages. Keys are issued by the :class:`Mailbox` instance

   with which they will be used and are only meaningful to that :class:`Mailbox`

   instance. A key continues to identify a message even if the corresponding

   message is modified, such as by replacing it with another message.

   Messages may be added to a :class:`Mailbox` instance using the set-like

   method :meth:`add` and removed using a ``del`` statement or the set-like

   methods :meth:`remove` and :meth:`discard`.

   :class:`Mailbox` interface semantics differ from dictionary semantics in some

   noteworthy ways. Each time a message is requested, a new representation

   (typically a :class:`Message` instance) is generated based upon the current

   state of the mailbox. Similarly, when a message is added to a

   :class:`Mailbox` instance, the provided message representation's contents are

   copied. In neither case is a reference to the message representation kept by

   the :class:`Mailbox` instance.

   The default :class:`Mailbox` iterator iterates over message representations,

   not keys as the default dictionary iterator does. Moreover, modification of a

   mailbox during iteration is safe and well-defined. Messages added to the

   mailbox after an iterator is created will not be seen by the

   iterator. Messages removed from the mailbox before the iterator yields them

   will be silently skipped, though using a key from an iterator may result in a

   :exc:`KeyError` exception if the corresponding message is subsequently

   removed.

   .. warning::

      Be very cautious when modifying mailboxes that might be simultaneously

      changed by some other process.  The safest mailbox format to use for such

      tasks is Maildir; try to avoid using single-file formats such as mbox for

      concurrent writing.  If you're modifying a mailbox, you *must* lock it by

      calling the :meth:`lock` and :meth:`unlock` methods *before* reading any

      messages in the file or making any changes by adding or deleting a

      message.  Failing to lock the mailbox runs the risk of losing messages or

      corrupting the entire mailbox.

   :class:`Mailbox` instances have the following methods:

   .. method:: add(message)

      Add *message* to the mailbox and return the key that has been assigned to

      it.

      Parameter *message* may be a :class:`Message` instance, an

      :class:`email.Message.Message` instance, a string, or a file-like object

      (which should be open in text mode). If *message* is an instance of the

      appropriate format-specific :class:`Message` subclass (e.g., if it's an

      :class:`mboxMessage` instance and this is an :class:`mbox` instance), its

      format-specific information is used. Otherwise, reasonable defaults for

      format-specific information are used.

   .. method:: remove(key)

               __delitem__(key)

               discard(key)

      Delete the message corresponding to *key* from the mailbox.

      If no such message exists, a :exc:`KeyError` exception is raised if the

      method was called as :meth:`remove` or :meth:`__delitem__` but no

      exception is raised if the method was called as :meth:`discard`. The

      behavior of :meth:`discard` may be preferred if the underlying mailbox

      format supports concurrent modification by other processes.

   .. method:: __setitem__(key, message)

      Replace the message corresponding to *key* with *message*. Raise a

      :exc:`KeyError` exception if no message already corresponds to *key*.

      As with :meth:`add`, parameter *message* may be a :class:`Message`

      instance, an :class:`email.Message.Message` instance, a string, or a

      file-like object (which should be open in text mode). If *message* is an

      instance of the appropriate format-specific :class:`Message` subclass

      (e.g., if it's an :class:`mboxMessage` instance and this is an

      :class:`mbox` instance), its format-specific information is

      used. Otherwise, the format-specific information of the message that

      currently corresponds to *key* is left unchanged.

   .. method:: iterkeys()

               keys()

      Return an iterator over all keys if called as :meth:`iterkeys` or return a

      list of keys if called as :meth:`keys`.

   .. method:: itervalues()

               __iter__()

               values()

      Return an iterator over representations of all messages if called as

      :meth:`itervalues` or :meth:`__iter__` or return a list of such

      representations if called as :meth:`values`. The messages are represented

      as instances of the appropriate format-specific :class:`Message` subclass

      unless a custom message factory was specified when the :class:`Mailbox`

      instance was initialized.

      .. note::

         The behavior of :meth:`__iter__` is unlike that of dictionaries, which

         iterate over keys.

   .. method:: iteritems()

               items()

      Return an iterator over (*key*, *message*) pairs, where *key* is a key and

      *message* is a message representation, if called as :meth:`iteritems` or

      return a list of such pairs if called as :meth:`items`. The messages are

      represented as instances of the appropriate format-specific

      :class:`Message` subclass unless a custom message factory was specified

      when the :class:`Mailbox` instance was initialized.

   .. method:: get(key[, default=None])

               __getitem__(key)

      Return a representation of the message corresponding to *key*. If no such

      message exists, *default* is returned if the method was called as

      :meth:`get` and a :exc:`KeyError` exception is raised if the method was

      called as :meth:`__getitem__`. The message is represented as an instance

      of the appropriate format-specific :class:`Message` subclass unless a

      custom message factory was specified when the :class:`Mailbox` instance

      was initialized.

   .. method:: get_message(key)

      Return a representation of the message corresponding to *key* as an

      instance of the appropriate format-specific :class:`Message` subclass, or

      raise a :exc:`KeyError` exception if no such message exists.

   .. method:: get_string(key)

      Return a string representation of the message corresponding to *key*, or

      raise a :exc:`KeyError` exception if no such message exists.

   .. method:: get_file(key)

      Return a file-like representation of the message corresponding to *key*,

      or raise a :exc:`KeyError` exception if no such message exists. The

      file-like object behaves as if open in binary mode. This file should be

      closed once it is no longer needed.

      .. note::

         Unlike other representations of messages, file-like representations are

         not necessarily independent of the :class:`Mailbox` instance that

         created them or of the underlying mailbox. More specific documentation

         is provided by each subclass.

   .. method:: has_key(key)

               __contains__(key)

      Return ``True`` if *key* corresponds to a message, ``False`` otherwise.

   .. method:: __len__()

      Return a count of messages in the mailbox.

   .. method:: clear()

      Delete all messages from the mailbox.

   .. method:: pop(key[, default])

      Return a representation of the message corresponding to *key* and delete

      the message. If no such message exists, return *default* if it was

      supplied or else raise a :exc:`KeyError` exception. The message is

      represented as an instance of the appropriate format-specific

      :class:`Message` subclass unless a custom message factory was specified

      when the :class:`Mailbox` instance was initialized.

   .. method:: popitem()

      Return an arbitrary (*key*, *message*) pair, where *key* is a key and

      *message* is a message representation, and delete the corresponding

      message. If the mailbox is empty, raise a :exc:`KeyError` exception. The

      message is represented as an instance of the appropriate format-specific

      :class:`Message` subclass unless a custom message factory was specified

      when the :class:`Mailbox` instance was initialized.

   .. method:: update(arg)

      Parameter *arg* should be a *key*-to-*message* mapping or an iterable of

      (*key*, *message*) pairs. Updates the mailbox so that, for each given

      *key* and *message*, the message corresponding to *key* is set to

      *message* as if by using :meth:`__setitem__`. As with :meth:`__setitem__`,

      each *key* must already correspond to a message in the mailbox or else a

      :exc:`KeyError` exception will be raised, so in general it is incorrect

      for *arg* to be a :class:`Mailbox` instance.

      .. note::

         Unlike with dictionaries, keyword arguments are not supported.

   .. method:: flush()

      Write any pending changes to the filesystem. For some :class:`Mailbox`

      subclasses, changes are always written immediately and :meth:`flush` does

      nothing, but you should still make a habit of calling this method.

   .. method:: lock()

      Acquire an exclusive advisory lock on the mailbox so that other processes

      know not to modify it. An :exc:`ExternalClashError` is raised if the lock

      is not available. The particular locking mechanisms used depend upon the

      mailbox format.  You should *always* lock the mailbox before making any

      modifications to its contents.

   .. method:: unlock()

      Release the lock on the mailbox, if any.

   .. method:: close()

      Flush the mailbox, unlock it if necessary, and close any open files. For

      some :class:`Mailbox` subclasses, this method does nothing.

.. _mailbox-maildir:

:class:`Maildir`

^^^^^^^^^^^^^^^^

.. class:: Maildir(dirname[, factory=rfc822.Message[, create=True]])

   A subclass of :class:`Mailbox` for mailboxes in Maildir format. Parameter

   *factory* is a callable object that accepts a file-like message representation

   (which behaves as if opened in binary mode) and returns a custom representation.

   If *factory* is ``None``, :class:`MaildirMessage` is used as the default message

   representation. If *create* is ``True``, the mailbox is created if it does not

   exist.

   It is for historical reasons that *factory* defaults to :class:`rfc822.Message`

   and that *dirname* is named as such rather than *path*. For a :class:`Maildir`

   instance that behaves like instances of other :class:`Mailbox` subclasses, set

   *factory* to ``None``.

   Maildir is a directory-based mailbox format invented for the qmail mail

   transfer agent and now widely supported by other programs. Messages in a

   Maildir mailbox are stored in separate files within a common directory

   structure. This design allows Maildir mailboxes to be accessed and modified

   by multiple unrelated programs without data corruption, so file locking is

   unnecessary.

   Maildir mailboxes contain three subdirectories, namely: :file:`tmp`,

   :file:`new`, and :file:`cur`. Messages are created momentarily in the

   :file:`tmp` subdirectory and then moved to the :file:`new` subdirectory to

   finalize delivery. A mail user agent may subsequently move the message to the

   :file:`cur` subdirectory and store information about the state of the message

   in a special "info" section appended to its file name.

   Folders of the style introduced by the Courier mail transfer agent are also

   supported. Any subdirectory of the main mailbox is considered a folder if

   ``'.'`` is the first character in its name. Folder names are represented by

   :class:`Maildir` without the leading ``'.'``. Each folder is itself a Maildir

   mailbox but should not contain other folders. Instead, a logical nesting is

   indicated using ``'.'`` to delimit levels, e.g., "Archived.2005.07".

   .. note::

      The Maildir specification requires the use of a colon (``':'``) in certain

      message file names. However, some operating systems do not permit this

      character in file names, If you wish to use a Maildir-like format on such

      an operating system, you should specify another character to use

      instead. The exclamation point (``'!'``) is a popular choice. For

      example::

         import mailbox

         mailbox.Maildir.colon = '!'

      The :attr:`colon` attribute may also be set on a per-instance basis.

   :class:`Maildir` instances have all of the methods of :class:`Mailbox` in

   addition to the following:

   .. method:: list_folders()

      Return a list of the names of all folders.

   .. method:: .et_folder(folder)

      Return a :class:`Maildir` instance representing the folder whose name is

      *folder*. A :exc:`NoSuchMailboxError` exception is raised if the folder

      does not exist.

   .. method:: add_folder(folder)

      Create a folder whose name is *folder* and return a :class:`Maildir`

      instance representing it.

   .. method:: remove_folder(folder)

      Delete the folder whose name is *folder*. If the folder contains any

      messages, a :exc:`NotEmptyError` exception will be raised and the folder

      will not be deleted.

   .. method:: clean()

      Delete temporary files from the mailbox that have not been accessed in the

      last 36 hours. The Maildir specification says that mail-reading programs

      should do this occasionally.

   Some :class:`Mailbox` methods implemented by :class:`Maildir` deserve special

   remarks:

   .. method:: add(message)

               __setitem__(key, message)

               update(arg)

      .. warning::

         These methods generate unique file names based upon the current process

         ID. When using multiple threads, undetected name clashes may occur and

         cause corruption of the mailbox unless threads are coordinated to avoid

         using these methods to manipulate the same mailbox simultaneously.

   .. method:: flush()

      All changes to Maildir mailboxes are immediately applied, so this method

      does nothing.

   .. method:: lock()

               unlock()

      Maildir mailboxes do not support (or require) locking, so these methods do

      nothing.

   .. method:: close()

      :class:`Maildir` instances do not keep any open files and the underlying

      mailboxes do not support locking, so this method does nothing.

   .. method:: get_file(key)

      Depending upon the host platform, it may not be possible to modify or

      remove the underlying message while the returned file remains open.

.. seealso::

   `maildir man page from qmail <http://www.qmail.org/man/man5/maildir.html>`_

      The original specification of the format.

   `Using maildir format <http://cr.yp.to/proto/maildir.html>`_

      Notes on Maildir by its inventor. Includes an updated name-creation scheme and

      details on "info" semantics.

   `maildir man page from Courier <http://www.courier-mta.org/maildir.html>`_

      Another specification of the format. Describes a common extension for supporting

      folders.

.. _mailbox-mbox:

:class:`mbox`

^^^^^^^^^^^^^

.. class:: mbox(path[, factory=None[, create=True]])

   A subclass of :class:`Mailbox` for mailboxes in mbox format. Parameter *factory*

   is a callable object that accepts a file-like message representation (which

   behaves as if opened in binary mode) and returns a custom representation. If

   *factory* is ``None``, :class:`mboxMessage` is used as the default message

   representation. If *create* is ``True``, the mailbox is created if it does not

   exist.

   The mbox format is the classic format for storing mail on Unix systems. All

   messages in an mbox mailbox are stored in a single file with the beginning of

   each message indicated by a line whose first five characters are "From ".

   Several variations of the mbox format exist to address perceived shortcomings in

   the original. In the interest of compatibility, :class:`mbox` implements the

   original format, which is sometimes referred to as :dfn:`mboxo`. This means that

   the :mailheader:`Content-Length` header, if present, is ignored and that any

   occurrences of "From " at the beginning of a line in a message body are

   transformed to ">From " when storing the message, although occurrences of ">From

   " are not transformed to "From " when reading the message.

   Some :class:`Mailbox` methods implemented by :class:`mbox` deserve special

   remarks:

   .. method:: get_file(key)

      Using the file after calling :meth:`flush` or :meth:`close` on the

      :class:`mbox` instance may yield unpredictable results or raise an

      exception.

   .. method:: lock()

               unlock()

      Three locking mechanisms are used---dot locking and, if available, the

      :cfunc:`flock` and :cfunc:`lockf` system calls.

.. seealso::

   `mbox man page from qmail <http://www.qmail.org/man/man5/mbox.html>`_

      A specification of the format and its variations.

   `mbox man page from tin <http://www.tin.org/bin/man.cgi?section=5&topic=mbox>`_

      Another specification of the format, with details on locking.

   `Configuring Netscape Mail on Unix: Why The Content-Length Format is Bad <http://www.jwz.org/doc/content-length.html>`_

      An argument for using the original mbox format rather than a variation.

   `"mbox" is a family of several mutually incompatible mailbox formats <http://homepages.tesco.net./~J.deBoynePollard/FGA/mail-mbox-formats.html>`_

      A history of mbox variations.

.. _mailbox-mh:

:class:`MH`

^^^^^^^^^^^

.. class:: MH(path[, factory=None[, create=True]])

   A subclass of :class:`Mailbox` for mailboxes in MH format. Parameter *factory*

   is a callable object that accepts a file-like message representation (which

   behaves as if opened in binary mode) and returns a custom representation. If

   *factory* is ``None``, :class:`MHMessage` is used as the default message

   representation. If *create* is ``True``, the mailbox is created if it does not

   exist.

   MH is a directory-based mailbox format invented for the MH Message Handling

   System, a mail user agent. Each message in an MH mailbox resides in its own

   file. An MH mailbox may contain other MH mailboxes (called :dfn:`folders`) in

   addition to messages. Folders may be nested indefinitely. MH mailboxes also

   support :dfn:`sequences`, which are named lists used to logically group

   messages without moving them to sub-folders. Sequences are defined in a file

   called :file:`.mh_sequences` in each folder.

   The :class:`MH` class manipulates MH mailboxes, but it does not attempt to

   emulate all of :program:`mh`'s behaviors. In particular, it does not modify

   and is not affected by the :file:`context` or :file:`.mh_profile` files that

   are used by :program:`mh` to store its state and configuration.

   :class:`MH` instances have all of the methods of :class:`Mailbox` in addition

   to the following:

   .. method:: list_folders()

      Return a list of the names of all folders.

   .. method:: get_folder(folder)

      Return an :class:`MH` instance representing the folder whose name is

      *folder*. A :exc:`NoSuchMailboxError` exception is raised if the folder

      does not exist.

   .. method:: add_folder(folder)

      Create a folder whose name is *folder* and return an :class:`MH` instance

      representing it.