:mod:`urllib2` --- extensible library for opening URLs

======================================================

.. module:: urllib2

   :synopsis: Next generation URL opening library.

.. moduleauthor:: Jeremy Hylton <jhylton@users.sourceforge.net>

.. sectionauthor:: Moshe Zadka <moshez@users.sourceforge.net>

.. note::

   The :mod:`urllib2` module has been split across several modules in

   Python 3.0 named :mod:`urllib.request` and :mod:`urllib.error`.

   The :term:`2to3` tool will automatically adapt imports when converting

   your sources to 3.0.

The :mod:`urllib2` module defines functions and classes which help in opening

URLs (mostly HTTP) in a complex world --- basic and digest authentication,

redirections, cookies and more.

The :mod:`urllib2` module defines the following functions:

.. function:: urlopen(url[, data][, timeout])

   Open the URL *url*, which can be either a string or a :class:`Request` object.

   *data* may be a string specifying additional data to send to the server, or

   ``None`` if no such data is needed.  Currently HTTP requests are the only ones

   that use *data*; the HTTP request will be a POST instead of a GET when the

   *data* parameter is provided.  *data* should be a buffer in the standard

   :mimetype:`application/x-www-form-urlencoded` format.  The

   :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and

   returns a string in this format.

   The optional *timeout* parameter specifies a timeout in seconds for blocking

   operations like the connection attempt (if not specified, the global default

   timeout setting will be used).  This actually only works for HTTP, HTTPS,

   FTP and FTPS connections.

   This function returns a file-like object with two additional methods:

   * :meth:`geturl` --- return the URL of the resource retrieved, commonly used to

     determine if a redirect was followed

   * :meth:`info` --- return the meta-information of the page, such as headers, in

     the form of an ``httplib.HTTPMessage`` instance

     (see `Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_)

   Raises :exc:`URLError` on errors.

   Note that ``None`` may be returned if no handler handles the request (though the

   default installed global :class:`OpenerDirector` uses :class:`UnknownHandler` to

   ensure this never happens).

   .. versionchanged:: 2.6

      *timeout* was added.

.. function:: install_opener(opener)

   Install an :class:`OpenerDirector` instance as the default global opener.

   Installing an opener is only necessary if you want urlopen to use that opener;

   otherwise, simply call :meth:`OpenerDirector.open` instead of :func:`urlopen`.

   The code does not check for a real :class:`OpenerDirector`, and any class with

   the appropriate interface will work.

.. function:: build_opener([handler, ...])

   Return an :class:`OpenerDirector` instance, which chains the handlers in the

   order given. *handler*\s can be either instances of :class:`BaseHandler`, or

   subclasses of :class:`BaseHandler` (in which case it must be possible to call

   the constructor without any parameters).  Instances of the following classes

   will be in front of the *handler*\s, unless the *handler*\s contain them,

   instances of them or subclasses of them: :class:`ProxyHandler`,

   :class:`UnknownHandler`, :class:`HTTPHandler`, :class:`HTTPDefaultErrorHandler`,

   :class:`HTTPRedirectHandler`, :class:`FTPHandler`, :class:`FileHandler`,

   :class:`HTTPErrorProcessor`.

   If the Python installation has SSL support (i.e., if the :mod:`ssl` module can be imported),

   :class:`HTTPSHandler` will also be added.

   Beginning in Python 2.3, a :class:`BaseHandler` subclass may also change its

   :attr:`handler_order` member variable to modify its position in the handlers

   list.

The following exceptions are raised as appropriate:

.. exception:: URLError

   The handlers raise this exception (or derived exceptions) when they run into a

   problem.  It is a subclass of :exc:`IOError`.

   .. attribute:: reason

      The reason for this error.  It can be a message string or another exception

      instance (:exc:`socket.error` for remote URLs, :exc:`OSError` for local

      URLs).

.. exception:: HTTPError

   Though being an exception (a subclass of :exc:`URLError`), an :exc:`HTTPError`

   can also function as a non-exceptional file-like return value (the same thing

   that :func:`urlopen` returns).  This is useful when handling exotic HTTP

   errors, such as requests for authentication.

   .. attribute:: code

      An HTTP status code as defined in `RFC 2616 <http://www.faqs.org/rfcs/rfc2616.html>`_. 

      This numeric value corresponds to a value found in the dictionary of

      codes as found in :attr:`BaseHTTPServer.BaseHTTPRequestHandler.responses`.

The following classes are provided:

.. class:: Request(url[, data][, headers][, origin_req_host][, unverifiable])

   This class is an abstraction of a URL request.

   *url* should be a string containing a valid URL.

   *data* may be a string specifying additional data to send to the server, or

   ``None`` if no such data is needed.  Currently HTTP requests are the only ones

   that use *data*; the HTTP request will be a POST instead of a GET when the

   *data* parameter is provided.  *data* should be a buffer in the standard

   :mimetype:`application/x-www-form-urlencoded` format.  The

   :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and

   returns a string in this format.

   *headers* should be a dictionary, and will be treated as if :meth:`add_header`

   was called with each key and value as arguments.  This is often used to "spoof"

   the ``User-Agent`` header, which is used by a browser to identify itself --

   some HTTP servers only allow requests coming from common browsers as opposed

   to scripts.  For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0

   (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib2`'s

   default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6).

   The final two arguments are only of interest for correct handling of third-party

   HTTP cookies:

   *origin_req_host* should be the request-host of the origin transaction, as

   defined by :rfc:`2965`.  It defaults to ``cookielib.request_host(self)``.  This

   is the host name or IP address of the original request that was initiated by the

   user.  For example, if the request is for an image in an HTML document, this

   should be the request-host of the request for the page containing the image.

   *unverifiable* should indicate whether the request is unverifiable, as defined

   by RFC 2965.  It defaults to False.  An unverifiable request is one whose URL

   the user did not have the option to approve.  For example, if the request is for

   an image in an HTML document, and the user had no option to approve the

   automatic fetching of the image, this should be true.

.. class:: OpenerDirector()

   The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained

   together. It manages the chaining of handlers, and recovery from errors.

.. class:: BaseHandler()

   This is the base class for all registered handlers --- and handles only the

   simple mechanics of registration.

.. class:: HTTPDefaultErrorHandler()

   A class which defines a default handler for HTTP error responses; all responses

   are turned into :exc:`HTTPError` exceptions.

.. class:: HTTPRedirectHandler()

   A class to handle redirections.

.. class:: HTTPCookieProcessor([cookiejar])

   A class to handle HTTP Cookies.

.. class:: ProxyHandler([proxies])

   Cause requests to go through a proxy. If *proxies* is given, it must be a

   dictionary mapping protocol names to URLs of proxies. The default is to read the

   list of proxies from the environment variables :envvar:`<protocol>_proxy`.

   To disable autodetected proxy pass an empty dictionary.

.. class:: HTTPPasswordMgr()

   Keep a database of  ``(realm, uri) -> (user, password)`` mappings.

.. class:: HTTPPasswordMgrWithDefaultRealm()

   Keep a database of  ``(realm, uri) -> (user, password)`` mappings. A realm of

   ``None`` is considered a catch-all realm, which is searched if no other realm

   fits.

.. class:: AbstractBasicAuthHandler([password_mgr])

   This is a mixin class that helps with HTTP authentication, both to the remote

   host and to a proxy. *password_mgr*, if given, should be something that is

   compatible with :class:`HTTPPasswordMgr`; refer to section

   :ref:`http-password-mgr` for information on the interface that must be

   supported.

.. class:: HTTPBasicAuthHandler([password_mgr])

   Handle authentication with the remote host. *password_mgr*, if given, should be

   something that is compatible with :class:`HTTPPasswordMgr`; refer to section

   :ref:`http-password-mgr` for information on the interface that must be

   supported.

.. class:: ProxyBasicAuthHandler([password_mgr])

   Handle authentication with the proxy. *password_mgr*, if given, should be

   something that is compatible with :class:`HTTPPasswordMgr`; refer to section

   :ref:`http-password-mgr` for information on the interface that must be

   supported.

.. class:: AbstractDigestAuthHandler([password_mgr])

   This is a mixin class that helps with HTTP authentication, both to the remote

   host and to a proxy. *password_mgr*, if given, should be something that is

   compatible with :class:`HTTPPasswordMgr`; refer to section

   :ref:`http-password-mgr` for information on the interface that must be

   supported.

.. class:: HTTPDigestAuthHandler([password_mgr])

   Handle authentication with the remote host. *password_mgr*, if given, should be

   something that is compatible with :class:`HTTPPasswordMgr`; refer to section

   :ref:`http-password-mgr` for information on the interface that must be

   supported.

.. class:: ProxyDigestAuthHandler([password_mgr])

   Handle authentication with the proxy. *password_mgr*, if given, should be

   something that is compatible with :class:`HTTPPasswordMgr`; refer to section

   :ref:`http-password-mgr` for information on the interface that must be

   supported.

.. class:: HTTPHandler()

   A class to handle opening of HTTP URLs.

.. class:: HTTPSHandler()

   A class to handle opening of HTTPS URLs.

.. class:: FileHandler()

   Open local files.

.. class:: FTPHandler()

   Open FTP URLs.

.. class:: CacheFTPHandler()

   Open FTP URLs, keeping a cache of open FTP connections to minimize delays.

.. class:: UnknownHandler()

   A catch-all class to handle unknown URLs.

.. _request-objects:

Request Objects

---------------

The following methods describe all of :class:`Request`'s public interface, and

so all must be overridden in subclasses.

.. method:: Request.add_data(data)

   Set the :class:`Request` data to *data*.  This is ignored by all handlers except

   HTTP handlers --- and there it should be a byte string, and will change the

   request to be ``POST`` rather than ``GET``.

.. method:: Request.get_method()

   Return a string indicating the HTTP request method.  This is only meaningful for

   HTTP requests, and currently always returns ``'GET'`` or ``'POST'``.

.. method:: Request.has_data()

   Return whether the instance has a non-\ ``None`` data.

.. method:: Request.get_data()

   Return the instance's data.

.. method:: Request.add_header(key, val)

   Add another header to the request.  Headers are currently ignored by all

   handlers except HTTP handlers, where they are added to the list of headers sent

   to the server.  Note that there cannot be more than one header with the same

   name, and later calls will overwrite previous calls in case the *key* collides.

   Currently, this is no loss of HTTP functionality, since all headers which have

   meaning when used more than once have a (header-specific) way of gaining the

   same functionality using only one header.

.. method:: Request.add_unredirected_header(key, header)

   Add a header that will not be added to a redirected request.

   .. versionadded:: 2.4

.. method:: Request.has_header(header)

   Return whether the instance has the named header (checks both regular and

   unredirected).

   .. versionadded:: 2.4

.. method:: Request.get_full_url()

   Return the URL given in the constructor.

.. method:: Request.get_type()

   Return the type of the URL --- also known as the scheme.

.. method:: Request.get_host()

   Return the host to which a connection will be made.

.. method:: Request.get_selector()

   Return the selector --- the part of the URL that is sent to the server.

.. method:: Request.set_proxy(host, type)

   Prepare the request by connecting to a proxy server. The *host* and *type* will

   replace those of the instance, and the instance's selector will be the original

   URL given in the constructor.

.. method:: Request.get_origin_req_host()

   Return the request-host of the origin transaction, as defined by :rfc:`2965`.

   See the documentation for the :class:`Request` constructor.

.. method:: Request.is_unverifiable()

   Return whether the request is unverifiable, as defined by RFC 2965. See the

   documentation for the :class:`Request` constructor.

.. _opener-director-objects:

OpenerDirector Objects

----------------------

:class:`OpenerDirector` instances have the following methods:

.. method:: OpenerDirector.add_handler(handler)

   *handler* should be an instance of :class:`BaseHandler`.  The following methods

   are searched, and added to the possible chains (note that HTTP errors are a

   special case).

   * :meth:`protocol_open` --- signal that the handler knows how to open *protocol*

     URLs.

   * :meth:`http_error_type` --- signal that the handler knows how to handle HTTP

     errors with HTTP error code *type*.

   * :meth:`protocol_error` --- signal that the handler knows how to handle errors

     from (non-\ ``http``) *protocol*.

   * :meth:`protocol_request` --- signal that the handler knows how to pre-process

     *protocol* requests.

   * :meth:`protocol_response` --- signal that the handler knows how to

     post-process *protocol* responses.

.. method:: OpenerDirector.open(url[, data][, timeout])

   Open the given *url* (which can be a request object or a string), optionally

   passing the given *data*. Arguments, return values and exceptions raised are

   the same as those of :func:`urlopen` (which simply calls the :meth:`open`

   method on the currently installed global :class:`OpenerDirector`).  The

   optional *timeout* parameter specifies a timeout in seconds for blocking

   operations like the connection attempt (if not specified, the global default

   timeout setting will be usedi). The timeout feature actually works only for

   HTTP, HTTPS, FTP and FTPS connections).

   .. versionchanged:: 2.6

      *timeout* was added.

.. method:: OpenerDirector.error(proto[, arg[, ...]])

   Handle an error of the given protocol.  This will call the registered error

   handlers for the given protocol with the given arguments (which are protocol

   specific).  The HTTP protocol is a special case which uses the HTTP response

   code to determine the specific error handler; refer to the :meth:`http_error_\*`

   methods of the handler classes.

   Return values and exceptions raised are the same as those of :func:`urlopen`.

OpenerDirector objects open URLs in three stages:

The order in which these methods are called within each stage is determined by

sorting the handler instances.

#. Every handler with a method named like :meth:`protocol_request` has that

   method called to pre-process the request.

#. Handlers with a method named like :meth:`protocol_open` are called to handle

   the request. This stage ends when a handler either returns a non-\ :const:`None`

   value (ie. a response), or raises an exception (usually :exc:`URLError`).

   Exceptions are allowed to propagate.

   In fact, the above algorithm is first tried for methods named

   :meth:`default_open`.  If all such methods return :const:`None`, the algorithm

   is repeated for methods named like :meth:`protocol_open`.  If all such methods

   return :const:`None`, the algorithm is repeated for methods named

   :meth:`unknown_open`.

   Note that the implementation of these methods may involve calls of the parent

   :class:`OpenerDirector` instance's :meth:`.open` and :meth:`.error` methods.

#. Every handler with a method named like :meth:`protocol_response` has that

   method called to post-process the response.

.. _base-handler-objects:

BaseHandler Objects

-------------------

:class:`BaseHandler` objects provide a couple of methods that are directly

useful, and others that are meant to be used by derived classes.  These are

intended for direct use:

.. method:: BaseHandler.add_parent(director)

   Add a director as parent.

.. method:: BaseHandler.close()

   Remove any parents.

The following members and methods should only be used by classes derived from

:class:`BaseHandler`.

.. note::

   The convention has been adopted that subclasses defining

   :meth:`protocol_request` or :meth:`protocol_response` methods are named

   :class:`\*Processor`; all others are named :class:`\*Handler`.

.. attribute:: BaseHandler.parent

   A valid :class:`OpenerDirector`, which can be used to open using a different

   protocol, or handle errors.

.. method:: BaseHandler.default_open(req)

   This method is *not* defined in :class:`BaseHandler`, but subclasses should

   define it if they want to catch all URLs.

   This method, if implemented, will be called by the parent

   :class:`OpenerDirector`.  It should return a file-like object as described in

   the return value of the :meth:`open` of :class:`OpenerDirector`, or ``None``.

   It should raise :exc:`URLError`, unless a truly exceptional thing happens (for

   example, :exc:`MemoryError` should not be mapped to :exc:`URLError`).

   This method will be called before any protocol-specific open method.

.. method:: BaseHandler.protocol_open(req)

   :noindex:

   This method is *not* defined in :class:`BaseHandler`, but subclasses should

   define it if they want to handle URLs with the given protocol.

   This method, if defined, will be called by the parent :class:`OpenerDirector`.

   Return values should be the same as for  :meth:`default_open`.

.. method:: BaseHandler.unknown_open(req)

   This method is *not* defined in :class:`BaseHandler`, but subclasses should

   define it if they want to catch all URLs with no specific registered handler to

   open it.