Doc/lib/liburllib.tex

\section{\module{urllib} ---
         Open an arbitrary object given by URL.}
\declaremodule{standard}{urllib}

\modulesynopsis{Open an arbitrary object given by URL (requires sockets).}

\index{WWW}
\index{World-Wide Web}
\index{URL}


This module provides a high-level interface for fetching data across
the World-Wide Web.  In particular, the \function{urlopen()} function
is similar to the built-in function \function{open()}, but accepts
Universal Resource Locators (URLs) instead of filenames.  Some
restrictions apply --- it can only open URLs for reading, and no seek
operations are available.

It defines the following public functions:

\begin{funcdesc}{urlopen}{url\optional{, data}}
Open a network object denoted by a URL for reading.  If the URL does
not have a scheme identifier, or if it has \file{file:} as its scheme
identifier, this opens a local file; otherwise it opens a socket to a
server somewhere on the network.  If the connection cannot be made, or
if the server returns an error code, the \exception{IOError} exception
is raised.  If all went well, a file-like object is returned.  This
supports the following methods: \method{read()}, \method{readline()},
\method{readlines()}, \method{fileno()}, \method{close()},
\method{info()} and \method{geturl()}.

Except for the \method{info()} and \method{geturl()} methods,
these methods have the same interface as for
file objects --- see section \ref{bltin-file-objects} in this
manual.  (It is not a built-in file object, however, so it can't be
used at those few places where a true built-in file object is
required.)

The \method{info()} method returns an instance of the class
\class{mimetools.Message} containing meta-information associated
with the URL.  When the method is HTTP, these headers are those
returned by the server at the head of the retrieved HTML page
(including Content-Length and Content-Type).  When the method is FTP,
a Content-Length header will be present if (as is now usual) the
server passed back a file length in response to the FTP retrieval
request.  When the method is local-file, returned headers will include
a Date representing the file's last-modified time, a Content-Length
giving file size, and a Content-Type containing a guess at the file's
type. See also the description of the
\refmodule{mimetools}\refstmodindex{mimetools} module.

The \method{geturl()} method returns the real URL of the page.  In
some cases, the HTTP server redirects a client to another URL.  The
\function{urlopen()} function handles this transparently, but in some
cases the caller needs to know which URL the client was redirected
to.  The \method{geturl()} method can be used to get at this
redirected URL.

If the \var{url} uses the \file{http:} scheme identifier, the optional
\var{data} argument may be given to specify a \code{POST} request
(normally the request type is \code{GET}).  The \var{data} argument
must in standard \file{application/x-www-form-urlencoded} format;
see the \function{urlencode()} function below.

\end{funcdesc}

\begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{, hook}}}
Copy a network object denoted by a URL to a local file, if necessary.
If the URL points to a local file, or a valid cached copy of the
object exists, the object is not copied.  Return a tuple
\code{(\var{filename}, \var{headers})} where \var{filename} is the
local file name under which the object can be found, and \var{headers}
is either \code{None} (for a local object) or whatever the
\method{info()} method of the object returned by \function{urlopen()}
returned (for a remote object, possibly cached).  Exceptions are the
same as for \function{urlopen()}.

The second argument, if present, specifies the file location to copy
to (if absent, the location will be a tempfile with a generated name).
The third argument, if present, is a hook function that will be called
once on establishment of the network connection and once after each
block read thereafter.  The hook will be passed three arguments; a
count of blocks transferred so far, a block size in bytes, and the
total size of the file.  The third argument may be \code{-1} on older
FTP servers which do not return a file size in response to a retrieval 
request.
\end{funcdesc}

\begin{funcdesc}{urlcleanup}{}
Clear the cache that may have been built up by previous calls to
\function{urlretrieve()}.
\end{funcdesc}

\begin{funcdesc}{quote}{string\optional{, safe}}
Replace special characters in \var{string} using the \samp{\%xx} escape.
Letters, digits, and the characters \character{_,.-} are never quoted.
The optional \var{safe} parameter specifies additional characters
that should not be quoted --- its default value is \code{'/'}.

Example: \code{quote('/\~connolly/')} yields \code{'/\%7econnolly/'}.
\end{funcdesc}

\begin{funcdesc}{quote_plus}{string\optional{, safe}}
Like \function{quote()}, but also replaces spaces by plus signs, as
required for quoting HTML form values.  Plus signs in the original
string are escaped unless they are included in \var{safe}.
\end{funcdesc}

\begin{funcdesc}{unquote}{string}
Replace \samp{\%xx} escapes by their single-character equivalent.

Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~connolly/'}.
\end{funcdesc}

\begin{funcdesc}{unquote_plus}{string}
Like \function{unquote()}, but also replaces plus signs by spaces, as
required for unquoting HTML form values.
\end{funcdesc}

\begin{funcdesc}{urlencode}{dict}
Convert a dictionary to a ``url-encoded'' string, suitable to pass to
\function{urlopen()} above as the optional \var{data} argument.  This
is useful to pass a dictionary of form fields to a \code{POST}
request.  The resulting string is a series of
\code{\var{key}=\var{value}} pairs separated by \character{\&}
characters, where both \var{key} and \var{value} are quoted using
\function{quote_plus()} above.
\end{funcdesc}

Restrictions:

\begin{itemize}

\item
Currently, only the following protocols are supported: HTTP, (versions
0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
\indexii{HTTP}{protocol}
\indexii{Gopher}{protocol}
\indexii{FTP}{protocol}

\item
The caching feature of \function{urlretrieve()} has been disabled
until I find the time to hack proper processing of Expiration time
headers.

\item
There should be a function to query whether a particular URL is in
the cache.

\item
For backward compatibility, if a URL appears to point to a local file
but the file can't be opened, the URL is re-interpreted using the FTP
protocol.  This can sometimes cause confusing error messages.

\item
The \function{urlopen()} and \function{urlretrieve()} functions can
cause arbitrarily long delays while waiting for a network connection
to be set up.  This means that it is difficult to build an interactive
web client using these functions without using threads.

\item
The data returned by \function{urlopen()} or \function{urlretrieve()}
is the raw data returned by the server.  This may be binary data
(e.g. an image), plain text or (for example) HTML\index{HTML}.  The
HTTP\indexii{HTTP}{protocol} protocol provides type information in the
reply header, which can be inspected by looking at the
\code{content-type} header.  For the Gopher\indexii{Gopher}{protocol}
protocol, type information is encoded in the URL; there is currently
no easy way to extract it.  If the returned data is HTML, you can use
the module \refmodule{htmllib}\refstmodindex{htmllib} to parse it.

\item
Although the \module{urllib} module contains (undocumented) routines
to parse and unparse URL strings, the recommended interface for URL
manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}.

\end{itemize}