blob: 49cd3c6f5d7cabf696224b544bb77d2ed0dc8477 [file] [log] [blame]
Fred Drake295da241998-08-10 19:42:37 +00001\section{\module{urllib} ---
Fred Drakedfca4dc2000-08-25 05:13:42 +00002 Open arbitrary resources by URL}
Fred Drakeb91e9341998-07-23 17:59:49 +00003
Fred Drake38e5d272000-04-03 20:13:55 +00004\declaremodule{standard}{urllib}
5\modulesynopsis{Open an arbitrary network resource by URL (requires sockets).}
Fred Drakeb91e9341998-07-23 17:59:49 +00006
Guido van Rossuma8db1df1995-02-16 16:29:46 +00007\index{WWW}
Fred Drake8ee679f2001-07-14 02:50:55 +00008\index{World Wide Web}
Guido van Rossum61d34f41995-02-27 17:51:51 +00009\index{URL}
Guido van Rossuma8db1df1995-02-16 16:29:46 +000010
Guido van Rossum86751151995-02-28 17:14:32 +000011
Guido van Rossuma8db1df1995-02-16 16:29:46 +000012This module provides a high-level interface for fetching data across
Fred Drake8ee679f2001-07-14 02:50:55 +000013the World Wide Web. In particular, the \function{urlopen()} function
Fred Drake6ef871c1998-03-12 06:52:05 +000014is similar to the built-in function \function{open()}, but accepts
15Universal Resource Locators (URLs) instead of filenames. Some
16restrictions apply --- it can only open URLs for reading, and no seek
17operations are available.
Guido van Rossuma8db1df1995-02-16 16:29:46 +000018
Fred Drakef5eaa2e1997-12-15 22:13:50 +000019It defines the following public functions:
Guido van Rossuma8db1df1995-02-16 16:29:46 +000020
Guido van Rossum0af2f631998-07-22 21:34:21 +000021\begin{funcdesc}{urlopen}{url\optional{, data}}
Guido van Rossuma8db1df1995-02-16 16:29:46 +000022Open a network object denoted by a URL for reading. If the URL does
Fred Drake6ef871c1998-03-12 06:52:05 +000023not have a scheme identifier, or if it has \file{file:} as its scheme
Guido van Rossuma8db1df1995-02-16 16:29:46 +000024identifier, this opens a local file; otherwise it opens a socket to a
25server somewhere on the network. If the connection cannot be made, or
Fred Drake6ef871c1998-03-12 06:52:05 +000026if the server returns an error code, the \exception{IOError} exception
27is raised. If all went well, a file-like object is returned. This
28supports the following methods: \method{read()}, \method{readline()},
Fred Drake1ec71cb1999-02-22 22:42:14 +000029\method{readlines()}, \method{fileno()}, \method{close()},
30\method{info()} and \method{geturl()}.
Guido van Rossum0af2f631998-07-22 21:34:21 +000031
Fred Drake1ec71cb1999-02-22 22:42:14 +000032Except for the \method{info()} and \method{geturl()} methods,
Guido van Rossum0af2f631998-07-22 21:34:21 +000033these methods have the same interface as for
Fred Drake6ef871c1998-03-12 06:52:05 +000034file objects --- see section \ref{bltin-file-objects} in this
35manual. (It is not a built-in file object, however, so it can't be
Guido van Rossum470be141995-03-17 16:07:09 +000036used at those few places where a true built-in file object is
37required.)
Guido van Rossuma8db1df1995-02-16 16:29:46 +000038
Fred Drake6ef871c1998-03-12 06:52:05 +000039The \method{info()} method returns an instance of the class
Guido van Rossum954b9ad1998-09-28 14:08:29 +000040\class{mimetools.Message} containing meta-information associated
41with the URL. When the method is HTTP, these headers are those
42returned by the server at the head of the retrieved HTML page
43(including Content-Length and Content-Type). When the method is FTP,
44a Content-Length header will be present if (as is now usual) the
45server passed back a file length in response to the FTP retrieval
Guido van Rossum88e0b5b2001-08-23 13:38:15 +000046request. A Content-Type header will be present if the MIME type can
47be guessed. When the method is local-file, returned headers will include
Guido van Rossum954b9ad1998-09-28 14:08:29 +000048a Date representing the file's last-modified time, a Content-Length
49giving file size, and a Content-Type containing a guess at the file's
50type. See also the description of the
Fred Drake1ec71cb1999-02-22 22:42:14 +000051\refmodule{mimetools}\refstmodindex{mimetools} module.
52
53The \method{geturl()} method returns the real URL of the page. In
54some cases, the HTTP server redirects a client to another URL. The
55\function{urlopen()} function handles this transparently, but in some
56cases the caller needs to know which URL the client was redirected
57to. The \method{geturl()} method can be used to get at this
58redirected URL.
Guido van Rossum0af2f631998-07-22 21:34:21 +000059
60If the \var{url} uses the \file{http:} scheme identifier, the optional
61\var{data} argument may be given to specify a \code{POST} request
62(normally the request type is \code{GET}). The \var{data} argument
Fred Drake51001332000-12-15 23:57:51 +000063must in standard \mimetype{application/x-www-form-urlencoded} format;
Guido van Rossum0af2f631998-07-22 21:34:21 +000064see the \function{urlencode()} function below.
65
Fred Drakeaef0e892000-08-31 17:23:35 +000066The \function{urlopen()} function works transparently with proxies
Fred Drake81c17352000-09-15 04:12:56 +000067which do not require authentication. In a \UNIX{} or Windows
Fred Drakeaef0e892000-08-31 17:23:35 +000068environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or
69\envvar{gopher_proxy} environment variables to a URL that identifies
70the proxy server before starting the Python interpreter. For example
71(the \character{\%} is the command prompt):
Fred Drake38e5d272000-04-03 20:13:55 +000072
73\begin{verbatim}
74% http_proxy="http://www.someproxy.com:3128"
75% export http_proxy
76% python
77...
78\end{verbatim}
79
80In a Macintosh environment, \function{urlopen()} will retrieve proxy
81information from Internet\index{Internet Config} Config.
82
Fred Drakeaef0e892000-08-31 17:23:35 +000083Proxies which require authentication for use are not currently
84supported; this is considered an implementation limitation.
Guido van Rossuma8db1df1995-02-16 16:29:46 +000085\end{funcdesc}
86
Fred Drake51001332000-12-15 23:57:51 +000087\begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{,
88 reporthook\optional{, data}}}}
Guido van Rossuma8db1df1995-02-16 16:29:46 +000089Copy a network object denoted by a URL to a local file, if necessary.
Guido van Rossum6c4f0031995-03-07 10:14:09 +000090If the URL points to a local file, or a valid cached copy of the
Fred Drake6ef871c1998-03-12 06:52:05 +000091object exists, the object is not copied. Return a tuple
92\code{(\var{filename}, \var{headers})} where \var{filename} is the
93local file name under which the object can be found, and \var{headers}
94is either \code{None} (for a local object) or whatever the
95\method{info()} method of the object returned by \function{urlopen()}
96returned (for a remote object, possibly cached). Exceptions are the
97same as for \function{urlopen()}.
Guido van Rossum954b9ad1998-09-28 14:08:29 +000098
99The second argument, if present, specifies the file location to copy
100to (if absent, the location will be a tempfile with a generated name).
101The third argument, if present, is a hook function that will be called
102once on establishment of the network connection and once after each
103block read thereafter. The hook will be passed three arguments; a
104count of blocks transferred so far, a block size in bytes, and the
Fred Drake09b29571998-10-01 20:43:13 +0000105total size of the file. The third argument may be \code{-1} on older
106FTP servers which do not return a file size in response to a retrieval
Guido van Rossum954b9ad1998-09-28 14:08:29 +0000107request.
Fred Drake9fa4d612000-08-24 01:06:40 +0000108
109If the \var{url} uses the \file{http:} scheme identifier, the optional
110\var{data} argument may be given to specify a \code{POST} request
111(normally the request type is \code{GET}). The \var{data} argument
Fred Drake51001332000-12-15 23:57:51 +0000112must in standard \mimetype{application/x-www-form-urlencoded} format;
Fred Drake9fa4d612000-08-24 01:06:40 +0000113see the \function{urlencode()} function below.
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000114\end{funcdesc}
115
116\begin{funcdesc}{urlcleanup}{}
117Clear the cache that may have been built up by previous calls to
Fred Drake6ef871c1998-03-12 06:52:05 +0000118\function{urlretrieve()}.
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000119\end{funcdesc}
120
Guido van Rossum0af2f631998-07-22 21:34:21 +0000121\begin{funcdesc}{quote}{string\optional{, safe}}
Fred Drake6ef871c1998-03-12 06:52:05 +0000122Replace special characters in \var{string} using the \samp{\%xx} escape.
123Letters, digits, and the characters \character{_,.-} are never quoted.
Guido van Rossum0af2f631998-07-22 21:34:21 +0000124The optional \var{safe} parameter specifies additional characters
Guido van Rossum61d34f41995-02-27 17:51:51 +0000125that should not be quoted --- its default value is \code{'/'}.
126
Fred Drake10853c92000-07-28 13:51:27 +0000127Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}.
Guido van Rossum8d40c841996-12-13 14:48:47 +0000128\end{funcdesc}
129
Guido van Rossum0af2f631998-07-22 21:34:21 +0000130\begin{funcdesc}{quote_plus}{string\optional{, safe}}
Fred Drake6ef871c1998-03-12 06:52:05 +0000131Like \function{quote()}, but also replaces spaces by plus signs, as
Guido van Rossum0af2f631998-07-22 21:34:21 +0000132required for quoting HTML form values. Plus signs in the original
133string are escaped unless they are included in \var{safe}.
Guido van Rossum61d34f41995-02-27 17:51:51 +0000134\end{funcdesc}
135
136\begin{funcdesc}{unquote}{string}
Guido van Rossum6c4f0031995-03-07 10:14:09 +0000137Replace \samp{\%xx} escapes by their single-character equivalent.
Guido van Rossum61d34f41995-02-27 17:51:51 +0000138
Fred Drake10853c92000-07-28 13:51:27 +0000139Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}.
Guido van Rossum61d34f41995-02-27 17:51:51 +0000140\end{funcdesc}
141
Guido van Rossum8d40c841996-12-13 14:48:47 +0000142\begin{funcdesc}{unquote_plus}{string}
Fred Drake6ef871c1998-03-12 06:52:05 +0000143Like \function{unquote()}, but also replaces plus signs by spaces, as
Guido van Rossum8d40c841996-12-13 14:48:47 +0000144required for unquoting HTML form values.
145\end{funcdesc}
146
Skip Montanaro4fda21b2001-01-28 21:18:16 +0000147\begin{funcdesc}{urlencode}{query\optional{, doseq}}
148Convert a mapping object or a sequence of two-element tuples to a
149``url-encoded'' string, suitable to pass to
Guido van Rossum0af2f631998-07-22 21:34:21 +0000150\function{urlopen()} above as the optional \var{data} argument. This
151is useful to pass a dictionary of form fields to a \code{POST}
Fred Drake09b29571998-10-01 20:43:13 +0000152request. The resulting string is a series of
153\code{\var{key}=\var{value}} pairs separated by \character{\&}
154characters, where both \var{key} and \var{value} are quoted using
Skip Montanaroeda28442001-01-24 06:36:06 +0000155\function{quote_plus()} above. If the optional parameter \var{doseq} is
156present and evaluates to true, individual \code{\var{key}=\var{value}} pairs
157are generated for each element of the sequence.
Skip Montanaro4fda21b2001-01-28 21:18:16 +0000158When a sequence of two-element tuples is used as the \var{query} argument,
159the first element of each tuple is a key and the second is a value. The
160order of parameters in the encoded string will match the order of parameter
161tuples in the sequence.
Guido van Rossum0af2f631998-07-22 21:34:21 +0000162\end{funcdesc}
163
Fred Drakedfca4dc2000-08-25 05:13:42 +0000164The public functions \function{urlopen()} and
165\function{urlretrieve()} create an instance of the
166\class{FancyURLopener} class and use it to perform their requested
167actions. To override this functionality, programmers can create a
168subclass of \class{URLopener} or \class{FancyURLopener}, then assign
169that an instance of that class to the
170\code{urllib._urlopener} variable before calling the desired function.
171For example, applications may want to specify a different
Fred Draked86038d2001-08-03 18:39:36 +0000172\mailheader{User-Agent} header than \class{URLopener} defines. This
173can be accomplished with the following code:
Fred Drake38e5d272000-04-03 20:13:55 +0000174
175\begin{verbatim}
176class AppURLopener(urllib.FancyURLopener):
177 def __init__(self, *args):
Fred Drake38e5d272000-04-03 20:13:55 +0000178 self.version = "App/1.7"
Fred Drake63bc2e02001-07-23 19:16:22 +0000179 urllib.FancyURLopener.__init__(self, *args)
Fred Drake38e5d272000-04-03 20:13:55 +0000180
Fred Drake6c160192000-05-30 14:39:45 +0000181urllib._urlopener = AppURLopener()
Fred Drake38e5d272000-04-03 20:13:55 +0000182\end{verbatim}
183
184\begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}}
185Base class for opening and reading URLs. Unless you need to support
186opening objects using schemes other than \file{http:}, \file{ftp:},
187\file{gopher:} or \file{file:}, you probably want to use
188\class{FancyURLopener}.
189
190By default, the \class{URLopener} class sends a
Fred Draked86038d2001-08-03 18:39:36 +0000191\mailheader{User-Agent} header of \samp{urllib/\var{VVV}}, where
Fred Drake38e5d272000-04-03 20:13:55 +0000192\var{VVV} is the \module{urllib} version number. Applications can
Fred Draked86038d2001-08-03 18:39:36 +0000193define their own \mailheader{User-Agent} header by subclassing
Fred Drake38e5d272000-04-03 20:13:55 +0000194\class{URLopener} or \class{FancyURLopener} and setting the instance
Fred Drakedfca4dc2000-08-25 05:13:42 +0000195attribute \member{version} to an appropriate string value before the
Fred Drake38e5d272000-04-03 20:13:55 +0000196\method{open()} method is called.
197
198Additional keyword parameters, collected in \var{x509}, are used for
199authentication with the \file{https:} scheme. The keywords
200\var{key_file} and \var{cert_file} are supported; both are needed to
201actually retrieve a resource at an \file{https:} URL.
202\end{classdesc}
203
204\begin{classdesc}{FancyURLopener}{...}
205\class{FancyURLopener} subclasses \class{URLopener} providing default
206handling for the following HTTP response codes: 301, 302 or 401. For
Fred Draked86038d2001-08-03 18:39:36 +0000207301 and 302 response codes, the \mailheader{Location} header is used to
Fred Drake38e5d272000-04-03 20:13:55 +0000208fetch the actual URL. For 401 response codes (authentication
Skip Montanaro04f1d372001-02-15 17:00:40 +0000209required), basic HTTP authentication is performed. For 301 and 302 response
210codes, recursion is bounded by the value of the \var{maxtries} attribute,
211which defaults 10.
Fred Drake38e5d272000-04-03 20:13:55 +0000212
213The parameters to the constructor are the same as those for
214\class{URLopener}.
Fred Drake47f11ce2001-04-12 20:26:49 +0000215
Fred Drake0aa811c2001-10-20 04:24:09 +0000216\note{When performing basic authentication, a
Fred Drake47f11ce2001-04-12 20:26:49 +0000217\class{FancyURLopener} instance calls its
218\method{prompt_user_passwd()} method. The default implementation asks
219the users for the required information on the controlling terminal. A
220subclass may override this method to support more appropriate behavior
Fred Drake0aa811c2001-10-20 04:24:09 +0000221if needed.}
Fred Drake38e5d272000-04-03 20:13:55 +0000222\end{classdesc}
223
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000224Restrictions:
225
226\begin{itemize}
227
228\item
229Currently, only the following protocols are supported: HTTP, (versions
2300.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
Fred Drake6ef871c1998-03-12 06:52:05 +0000231\indexii{HTTP}{protocol}
232\indexii{Gopher}{protocol}
233\indexii{FTP}{protocol}
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000234
235\item
Fred Drake6ef871c1998-03-12 06:52:05 +0000236The caching feature of \function{urlretrieve()} has been disabled
237until I find the time to hack proper processing of Expiration time
238headers.
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000239
240\item
Guido van Rossum6c4f0031995-03-07 10:14:09 +0000241There should be a function to query whether a particular URL is in
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000242the cache.
243
244\item
245For backward compatibility, if a URL appears to point to a local file
246but the file can't be opened, the URL is re-interpreted using the FTP
247protocol. This can sometimes cause confusing error messages.
248
249\item
Fred Drake6ef871c1998-03-12 06:52:05 +0000250The \function{urlopen()} and \function{urlretrieve()} functions can
251cause arbitrarily long delays while waiting for a network connection
252to be set up. This means that it is difficult to build an interactive
Fred Drake8ee679f2001-07-14 02:50:55 +0000253Web client using these functions without using threads.
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000254
255\item
Fred Drake6ef871c1998-03-12 06:52:05 +0000256The data returned by \function{urlopen()} or \function{urlretrieve()}
257is the raw data returned by the server. This may be binary data
Fred Drake1ec71cb1999-02-22 22:42:14 +0000258(e.g. an image), plain text or (for example) HTML\index{HTML}. The
259HTTP\indexii{HTTP}{protocol} protocol provides type information in the
260reply header, which can be inspected by looking at the
Fred Draked86038d2001-08-03 18:39:36 +0000261\mailheader{Content-Type} header. For the
262Gopher\indexii{Gopher}{protocol} protocol, type information is encoded
263in the URL; there is currently no easy way to extract it. If the
264returned data is HTML, you can use the module
265\refmodule{htmllib}\refstmodindex{htmllib} to parse it.
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000266
267\item
Fred Drake81c17352000-09-15 04:12:56 +0000268This module does not support the use of proxies which require
269authentication. This may be implemented in the future.
270
271\item
Fred Drake6ef871c1998-03-12 06:52:05 +0000272Although the \module{urllib} module contains (undocumented) routines
273to parse and unparse URL strings, the recommended interface for URL
Fred Drake1ec71cb1999-02-22 22:42:14 +0000274manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}.
Guido van Rossuma8db1df1995-02-16 16:29:46 +0000275
276\end{itemize}
Fred Drake38e5d272000-04-03 20:13:55 +0000277
278
279\subsection{URLopener Objects \label{urlopener-objs}}
280\sectionauthor{Skip Montanaro}{skip@mojam.com}
281
282\class{URLopener} and \class{FancyURLopener} objects have the
Fred Drakedfca4dc2000-08-25 05:13:42 +0000283following attributes.
Fred Drake38e5d272000-04-03 20:13:55 +0000284
Fred Drakedfca4dc2000-08-25 05:13:42 +0000285\begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}}
Fred Drake38e5d272000-04-03 20:13:55 +0000286Open \var{fullurl} using the appropriate protocol. This method sets
287up cache and proxy information, then calls the appropriate open method with
288its input arguments. If the scheme is not recognized,
289\method{open_unknown()} is called. The \var{data} argument
290has the same meaning as the \var{data} argument of \function{urlopen()}.
291\end{methoddesc}
292
Fred Drakedfca4dc2000-08-25 05:13:42 +0000293\begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}}
Fred Drake38e5d272000-04-03 20:13:55 +0000294Overridable interface to open unknown URL types.
295\end{methoddesc}
296
Fred Drakedfca4dc2000-08-25 05:13:42 +0000297\begin{methoddesc}[URLopener]{retrieve}{url\optional{,
298 filename\optional{,
299 reporthook\optional{, data}}}}
Fred Drake38e5d272000-04-03 20:13:55 +0000300Retrieves the contents of \var{url} and places it in \var{filename}. The
301return value is a tuple consisting of a local filename and either a
302\class{mimetools.Message} object containing the response headers (for remote
303URLs) or None (for local URLs). The caller must then open and read the
304contents of \var{filename}. If \var{filename} is not given and the URL
305refers to a local file, the input filename is returned. If the URL is
306non-local and \var{filename} is not given, the filename is the output of
307\function{tempfile.mktemp()} with a suffix that matches the suffix of the last
308path component of the input URL. If \var{reporthook} is given, it must be
309a function accepting three numeric parameters. It will be called after each
310chunk of data is read from the network. \var{reporthook} is ignored for
311local URLs.
Fred Drake9fa4d612000-08-24 01:06:40 +0000312
313If the \var{url} uses the \file{http:} scheme identifier, the optional
314\var{data} argument may be given to specify a \code{POST} request
315(normally the request type is \code{GET}). The \var{data} argument
Fred Drake51001332000-12-15 23:57:51 +0000316must in standard \mimetype{application/x-www-form-urlencoded} format;
Fred Drake9fa4d612000-08-24 01:06:40 +0000317see the \function{urlencode()} function below.
Fred Drake38e5d272000-04-03 20:13:55 +0000318\end{methoddesc}
319
Fred Drakedfca4dc2000-08-25 05:13:42 +0000320\begin{memberdesc}[URLopener]{version}
321Variable that specifies the user agent of the opener object. To get
322\refmodule{urllib} to tell servers that it is a particular user agent,
323set this in a subclass as a class variable or in the constructor
324before calling the base constructor.
325\end{memberdesc}
326
Fred Drake47f11ce2001-04-12 20:26:49 +0000327The \class{FancyURLopener} class offers one additional method that
328should be overloaded to provide the appropriate behavior:
329
330\begin{methoddesc}[FancyURLopener]{prompt_user_passwd}{host, realm}
331Return information needed to authenticate the user at the given host
332in the specified security realm. The return value should be a tuple,
333\code{(\var{user}, \var{password})}, which can be used for basic
334authentication.
335
336The implementation prompts for this information on the terminal; an
337application should override this method to use an appropriate
338interaction model in the local environment.
339\end{methoddesc}
340
Fred Drake38e5d272000-04-03 20:13:55 +0000341
342\subsection{Examples}
343\nodename{Urllib Examples}
344
345Here is an example session that uses the \samp{GET} method to retrieve
346a URL containing parameters:
347
348\begin{verbatim}
349>>> import urllib
350>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
351>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
352>>> print f.read()
353\end{verbatim}
354
355The following example uses the \samp{POST} method instead:
356
357\begin{verbatim}
358>>> import urllib
359>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
360>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
361>>> print f.read()
362\end{verbatim}