Fred Drake | 295da24 | 1998-08-10 19:42:37 +0000 | [diff] [blame] | 1 | \section{\module{urllib} --- |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 2 | Open arbitrary resources by URL} |
Fred Drake | b91e934 | 1998-07-23 17:59:49 +0000 | [diff] [blame] | 3 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 4 | \declaremodule{standard}{urllib} |
| 5 | \modulesynopsis{Open an arbitrary network resource by URL (requires sockets).} |
Fred Drake | b91e934 | 1998-07-23 17:59:49 +0000 | [diff] [blame] | 6 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 7 | \index{WWW} |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 8 | \index{World Wide Web} |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 9 | \index{URL} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 10 | |
Guido van Rossum | 8675115 | 1995-02-28 17:14:32 +0000 | [diff] [blame] | 11 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 12 | This module provides a high-level interface for fetching data across |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 13 | the World Wide Web. In particular, the \function{urlopen()} function |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 14 | is similar to the built-in function \function{open()}, but accepts |
| 15 | Universal Resource Locators (URLs) instead of filenames. Some |
| 16 | restrictions apply --- it can only open URLs for reading, and no seek |
| 17 | operations are available. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 18 | |
Fred Drake | f5eaa2e | 1997-12-15 22:13:50 +0000 | [diff] [blame] | 19 | It defines the following public functions: |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 20 | |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 21 | \begin{funcdesc}{urlopen}{url\optional{, data\optional{, proxies}}} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 22 | Open a network object denoted by a URL for reading. If the URL does |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 23 | not have a scheme identifier, or if it has \file{file:} as its scheme |
Brett Cannon | a2f8737 | 2003-04-29 04:11:12 +0000 | [diff] [blame] | 24 | identifier, this opens a local file (without universal newlines); |
| 25 | otherwise it opens a socket to a server somewhere on the network. If |
Georg Brandl | 124a4e5 | 2006-02-20 21:26:18 +0000 | [diff] [blame] | 26 | the connection cannot be made |
Brett Cannon | a2f8737 | 2003-04-29 04:11:12 +0000 | [diff] [blame] | 27 | the \exception{IOError} exception is raised. If all went well, a |
| 28 | file-like object is returned. This supports the following methods: |
| 29 | \method{read()}, \method{readline()}, \method{readlines()}, \method{fileno()}, |
| 30 | \method{close()}, \method{info()} and \method{geturl()}. It also has |
| 31 | proper support for the iterator protocol. |
Fred Drake | 504ca68 | 2004-03-25 16:51:12 +0000 | [diff] [blame] | 32 | One caveat: the \method{read()} method, if the size argument is |
| 33 | omitted or negative, may not read until the end of the data stream; |
| 34 | there is no good way to determine that the entire stream from a socket |
| 35 | has been read in the general case. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 36 | |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 37 | Except for the \method{info()} and \method{geturl()} methods, |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 38 | these methods have the same interface as for |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 39 | file objects --- see section \ref{bltin-file-objects} in this |
| 40 | manual. (It is not a built-in file object, however, so it can't be |
Guido van Rossum | 470be14 | 1995-03-17 16:07:09 +0000 | [diff] [blame] | 41 | used at those few places where a true built-in file object is |
| 42 | required.) |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 43 | |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 44 | The \method{info()} method returns an instance of the class |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 45 | \class{mimetools.Message} containing meta-information associated |
| 46 | with the URL. When the method is HTTP, these headers are those |
| 47 | returned by the server at the head of the retrieved HTML page |
| 48 | (including Content-Length and Content-Type). When the method is FTP, |
| 49 | a Content-Length header will be present if (as is now usual) the |
| 50 | server passed back a file length in response to the FTP retrieval |
Guido van Rossum | 88e0b5b | 2001-08-23 13:38:15 +0000 | [diff] [blame] | 51 | request. A Content-Type header will be present if the MIME type can |
| 52 | be guessed. When the method is local-file, returned headers will include |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 53 | a Date representing the file's last-modified time, a Content-Length |
| 54 | giving file size, and a Content-Type containing a guess at the file's |
| 55 | type. See also the description of the |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 56 | \refmodule{mimetools}\refstmodindex{mimetools} module. |
| 57 | |
| 58 | The \method{geturl()} method returns the real URL of the page. In |
| 59 | some cases, the HTTP server redirects a client to another URL. The |
| 60 | \function{urlopen()} function handles this transparently, but in some |
| 61 | cases the caller needs to know which URL the client was redirected |
| 62 | to. The \method{geturl()} method can be used to get at this |
| 63 | redirected URL. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 64 | |
| 65 | If the \var{url} uses the \file{http:} scheme identifier, the optional |
| 66 | \var{data} argument may be given to specify a \code{POST} request |
| 67 | (normally the request type is \code{GET}). The \var{data} argument |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 68 | must be in standard \mimetype{application/x-www-form-urlencoded} format; |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 69 | see the \function{urlencode()} function below. |
| 70 | |
Fred Drake | aef0e89 | 2000-08-31 17:23:35 +0000 | [diff] [blame] | 71 | The \function{urlopen()} function works transparently with proxies |
Fred Drake | 81c1735 | 2000-09-15 04:12:56 +0000 | [diff] [blame] | 72 | which do not require authentication. In a \UNIX{} or Windows |
Fred Drake | aef0e89 | 2000-08-31 17:23:35 +0000 | [diff] [blame] | 73 | environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or |
| 74 | \envvar{gopher_proxy} environment variables to a URL that identifies |
| 75 | the proxy server before starting the Python interpreter. For example |
| 76 | (the \character{\%} is the command prompt): |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 77 | |
| 78 | \begin{verbatim} |
| 79 | % http_proxy="http://www.someproxy.com:3128" |
| 80 | % export http_proxy |
| 81 | % python |
| 82 | ... |
| 83 | \end{verbatim} |
| 84 | |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 85 | In a Windows environment, if no proxy environment variables are set, |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 86 | proxy settings are obtained from the registry's Internet Settings |
| 87 | section. |
| 88 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 89 | In a Macintosh environment, \function{urlopen()} will retrieve proxy |
| 90 | information from Internet\index{Internet Config} Config. |
| 91 | |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 92 | Alternatively, the optional \var{proxies} argument may be used to |
| 93 | explicitly specify proxies. It must be a dictionary mapping scheme |
| 94 | names to proxy URLs, where an empty dictionary causes no proxies to be |
| 95 | used, and \code{None} (the default value) causes environmental proxy |
| 96 | settings to be used as discussed above. For example: |
| 97 | |
| 98 | \begin{verbatim} |
| 99 | # Use http://www.someproxy.com:3128 for http proxying |
Andrew M. Kuchling | 597bd60 | 2005-06-01 15:26:24 +0000 | [diff] [blame] | 100 | proxies = {'http': 'http://www.someproxy.com:3128'} |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 101 | filehandle = urllib.urlopen(some_url, proxies=proxies) |
| 102 | # Don't use any proxies |
| 103 | filehandle = urllib.urlopen(some_url, proxies={}) |
| 104 | # Use proxies from environment - both versions are equivalent |
| 105 | filehandle = urllib.urlopen(some_url, proxies=None) |
| 106 | filehandle = urllib.urlopen(some_url) |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 107 | \end{verbatim} |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 108 | |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 109 | The \function{urlopen()} function does not support explicit proxy |
| 110 | specification. If you need to override environmental proxy settings, |
| 111 | use \class{URLopener}, or a subclass such as \class{FancyURLopener}. |
| 112 | |
Fred Drake | aef0e89 | 2000-08-31 17:23:35 +0000 | [diff] [blame] | 113 | Proxies which require authentication for use are not currently |
| 114 | supported; this is considered an implementation limitation. |
Fred Drake | a2c2595 | 2002-04-04 20:58:02 +0000 | [diff] [blame] | 115 | |
| 116 | \versionchanged[Added the \var{proxies} support]{2.3} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 117 | \end{funcdesc} |
| 118 | |
Fred Drake | 5100133 | 2000-12-15 23:57:51 +0000 | [diff] [blame] | 119 | \begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{, |
| 120 | reporthook\optional{, data}}}} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 121 | Copy a network object denoted by a URL to a local file, if necessary. |
Guido van Rossum | 6c4f003 | 1995-03-07 10:14:09 +0000 | [diff] [blame] | 122 | If the URL points to a local file, or a valid cached copy of the |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 123 | object exists, the object is not copied. Return a tuple |
| 124 | \code{(\var{filename}, \var{headers})} where \var{filename} is the |
| 125 | local file name under which the object can be found, and \var{headers} |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 126 | is whatever the \method{info()} method of the object returned by |
| 127 | \function{urlopen()} returned (for a remote object, possibly cached). |
| 128 | Exceptions are the same as for \function{urlopen()}. |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 129 | |
| 130 | The second argument, if present, specifies the file location to copy |
| 131 | to (if absent, the location will be a tempfile with a generated name). |
| 132 | The third argument, if present, is a hook function that will be called |
| 133 | once on establishment of the network connection and once after each |
| 134 | block read thereafter. The hook will be passed three arguments; a |
| 135 | count of blocks transferred so far, a block size in bytes, and the |
Fred Drake | 09b2957 | 1998-10-01 20:43:13 +0000 | [diff] [blame] | 136 | total size of the file. The third argument may be \code{-1} on older |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 137 | FTP servers which do not return a file size in response to a retrieval |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 138 | request. |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 139 | |
| 140 | If the \var{url} uses the \file{http:} scheme identifier, the optional |
| 141 | \var{data} argument may be given to specify a \code{POST} request |
| 142 | (normally the request type is \code{GET}). The \var{data} argument |
Fred Drake | 5100133 | 2000-12-15 23:57:51 +0000 | [diff] [blame] | 143 | must in standard \mimetype{application/x-www-form-urlencoded} format; |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 144 | see the \function{urlencode()} function below. |
Georg Brandl | b925602 | 2005-08-24 18:46:39 +0000 | [diff] [blame] | 145 | |
| 146 | \versionchanged[ |
| 147 | \function{urlretrieve()} will raise \exception{ContentTooShortError} |
| 148 | when it detects that the amount of data available |
| 149 | was less than the expected amount (which is the size reported by a |
| 150 | \var{Content-Length} header). This can occur, for example, when the |
| 151 | download is interrupted. |
| 152 | |
| 153 | The \var{Content-Length} is treated as a lower bound: if there's more data |
| 154 | to read, urlretrieve reads more data, but if less data is available, |
| 155 | it raises the exception. |
| 156 | |
| 157 | You can still retrieve the downloaded data in this case, it is stored |
| 158 | in the \member{content} attribute of the exception instance. |
| 159 | |
| 160 | If no \var{Content-Length} header was supplied, urlretrieve can |
| 161 | not check the size of the data it has downloaded, and just returns it. |
| 162 | In this case you just have to assume that the download was successful]{2.5} |
| 163 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 164 | \end{funcdesc} |
| 165 | |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 166 | \begin{datadesc}{_urlopener} |
| 167 | The public functions \function{urlopen()} and |
| 168 | \function{urlretrieve()} create an instance of the |
| 169 | \class{FancyURLopener} class and use it to perform their requested |
| 170 | actions. To override this functionality, programmers can create a |
| 171 | subclass of \class{URLopener} or \class{FancyURLopener}, then assign |
Fred Drake | 4f4dbef | 2003-08-27 15:11:40 +0000 | [diff] [blame] | 172 | an instance of that class to the |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 173 | \code{urllib._urlopener} variable before calling the desired function. |
| 174 | For example, applications may want to specify a different |
| 175 | \mailheader{User-Agent} header than \class{URLopener} defines. This |
| 176 | can be accomplished with the following code: |
| 177 | |
| 178 | \begin{verbatim} |
| 179 | import urllib |
| 180 | |
| 181 | class AppURLopener(urllib.FancyURLopener): |
Skip Montanaro | 8e97fbf | 2005-03-09 02:57:26 +0000 | [diff] [blame] | 182 | version = "App/1.7" |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 183 | |
| 184 | urllib._urlopener = AppURLopener() |
| 185 | \end{verbatim} |
| 186 | \end{datadesc} |
| 187 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 188 | \begin{funcdesc}{urlcleanup}{} |
| 189 | Clear the cache that may have been built up by previous calls to |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 190 | \function{urlretrieve()}. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 191 | \end{funcdesc} |
| 192 | |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 193 | \begin{funcdesc}{quote}{string\optional{, safe}} |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 194 | Replace special characters in \var{string} using the \samp{\%xx} escape. |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 195 | Letters, digits, and the characters \character{_.-} are never quoted. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 196 | The optional \var{safe} parameter specifies additional characters |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 197 | that should not be quoted --- its default value is \code{'/'}. |
| 198 | |
Fred Drake | 10853c9 | 2000-07-28 13:51:27 +0000 | [diff] [blame] | 199 | Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}. |
Guido van Rossum | 8d40c84 | 1996-12-13 14:48:47 +0000 | [diff] [blame] | 200 | \end{funcdesc} |
| 201 | |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 202 | \begin{funcdesc}{quote_plus}{string\optional{, safe}} |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 203 | Like \function{quote()}, but also replaces spaces by plus signs, as |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 204 | required for quoting HTML form values. Plus signs in the original |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 205 | string are escaped unless they are included in \var{safe}. It also |
| 206 | does not have \var{safe} default to \code{'/'}. |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 207 | \end{funcdesc} |
| 208 | |
| 209 | \begin{funcdesc}{unquote}{string} |
Guido van Rossum | 6c4f003 | 1995-03-07 10:14:09 +0000 | [diff] [blame] | 210 | Replace \samp{\%xx} escapes by their single-character equivalent. |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 211 | |
Fred Drake | 10853c9 | 2000-07-28 13:51:27 +0000 | [diff] [blame] | 212 | Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}. |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 213 | \end{funcdesc} |
| 214 | |
Guido van Rossum | 8d40c84 | 1996-12-13 14:48:47 +0000 | [diff] [blame] | 215 | \begin{funcdesc}{unquote_plus}{string} |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 216 | Like \function{unquote()}, but also replaces plus signs by spaces, as |
Guido van Rossum | 8d40c84 | 1996-12-13 14:48:47 +0000 | [diff] [blame] | 217 | required for unquoting HTML form values. |
| 218 | \end{funcdesc} |
| 219 | |
Skip Montanaro | 4fda21b | 2001-01-28 21:18:16 +0000 | [diff] [blame] | 220 | \begin{funcdesc}{urlencode}{query\optional{, doseq}} |
| 221 | Convert a mapping object or a sequence of two-element tuples to a |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 222 | ``url-encoded'' string, suitable to pass to |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 223 | \function{urlopen()} above as the optional \var{data} argument. This |
| 224 | is useful to pass a dictionary of form fields to a \code{POST} |
Fred Drake | 09b2957 | 1998-10-01 20:43:13 +0000 | [diff] [blame] | 225 | request. The resulting string is a series of |
| 226 | \code{\var{key}=\var{value}} pairs separated by \character{\&} |
| 227 | characters, where both \var{key} and \var{value} are quoted using |
Skip Montanaro | eda2844 | 2001-01-24 06:36:06 +0000 | [diff] [blame] | 228 | \function{quote_plus()} above. If the optional parameter \var{doseq} is |
| 229 | present and evaluates to true, individual \code{\var{key}=\var{value}} pairs |
| 230 | are generated for each element of the sequence. |
Skip Montanaro | 4fda21b | 2001-01-28 21:18:16 +0000 | [diff] [blame] | 231 | When a sequence of two-element tuples is used as the \var{query} argument, |
| 232 | the first element of each tuple is a key and the second is a value. The |
| 233 | order of parameters in the encoded string will match the order of parameter |
| 234 | tuples in the sequence. |
Fred Drake | d859d47 | 2003-04-24 16:22:47 +0000 | [diff] [blame] | 235 | The \refmodule{cgi} module provides the functions |
| 236 | \function{parse_qs()} and \function{parse_qsl()} which are used to |
| 237 | parse query strings into Python data structures. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 238 | \end{funcdesc} |
| 239 | |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 240 | \begin{funcdesc}{pathname2url}{path} |
| 241 | Convert the pathname \var{path} from the local syntax for a path to |
| 242 | the form used in the path component of a URL. This does not produce a |
| 243 | complete URL. The return value will already be quoted using the |
| 244 | \function{quote()} function. |
| 245 | \end{funcdesc} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 246 | |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 247 | \begin{funcdesc}{url2pathname}{path} |
| 248 | Convert the path component \var{path} from an encoded URL to the local |
| 249 | syntax for a path. This does not accept a complete URL. This |
| 250 | function uses \function{unquote()} to decode \var{path}. |
| 251 | \end{funcdesc} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 252 | |
| 253 | \begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}} |
| 254 | Base class for opening and reading URLs. Unless you need to support |
| 255 | opening objects using schemes other than \file{http:}, \file{ftp:}, |
| 256 | \file{gopher:} or \file{file:}, you probably want to use |
| 257 | \class{FancyURLopener}. |
| 258 | |
| 259 | By default, the \class{URLopener} class sends a |
Fred Drake | d86038d | 2001-08-03 18:39:36 +0000 | [diff] [blame] | 260 | \mailheader{User-Agent} header of \samp{urllib/\var{VVV}}, where |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 261 | \var{VVV} is the \module{urllib} version number. Applications can |
Fred Drake | d86038d | 2001-08-03 18:39:36 +0000 | [diff] [blame] | 262 | define their own \mailheader{User-Agent} header by subclassing |
Skip Montanaro | 8e97fbf | 2005-03-09 02:57:26 +0000 | [diff] [blame] | 263 | \class{URLopener} or \class{FancyURLopener} and setting the class |
| 264 | attribute \member{version} to an appropriate string value in the |
| 265 | subclass definition. |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 266 | |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 267 | The optional \var{proxies} parameter should be a dictionary mapping |
| 268 | scheme names to proxy URLs, where an empty dictionary turns proxies |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 269 | off completely. Its default value is \code{None}, in which case |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 270 | environmental proxy settings will be used if present, as discussed in |
| 271 | the definition of \function{urlopen()}, above. |
| 272 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 273 | Additional keyword parameters, collected in \var{x509}, are used for |
| 274 | authentication with the \file{https:} scheme. The keywords |
| 275 | \var{key_file} and \var{cert_file} are supported; both are needed to |
| 276 | actually retrieve a resource at an \file{https:} URL. |
Georg Brandl | 124a4e5 | 2006-02-20 21:26:18 +0000 | [diff] [blame] | 277 | |
| 278 | \class{URLopener} objects will raise an \exception{IOError} exception |
| 279 | if the server returns an error code. |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 280 | \end{classdesc} |
| 281 | |
| 282 | \begin{classdesc}{FancyURLopener}{...} |
| 283 | \class{FancyURLopener} subclasses \class{URLopener} providing default |
Martin v. Löwis | 162f081 | 2003-07-12 07:33:32 +0000 | [diff] [blame] | 284 | handling for the following HTTP response codes: 301, 302, 303, 307 and |
| 285 | 401. For the 30x response codes listed above, the |
| 286 | \mailheader{Location} header is used to fetch the actual URL. For 401 |
| 287 | response codes (authentication required), basic HTTP authentication is |
| 288 | performed. For the 30x response codes, recursion is bounded by the |
| 289 | value of the \var{maxtries} attribute, which defaults to 10. |
Raymond Hettinger | 024aaa1 | 2003-04-24 15:32:12 +0000 | [diff] [blame] | 290 | |
Georg Brandl | 124a4e5 | 2006-02-20 21:26:18 +0000 | [diff] [blame] | 291 | For all other response codes, the method \method{http_error_default()} |
| 292 | is called which you can override in subclasses to handle the error |
| 293 | appropriately. |
| 294 | |
Raymond Hettinger | 024aaa1 | 2003-04-24 15:32:12 +0000 | [diff] [blame] | 295 | \note{According to the letter of \rfc{2616}, 301 and 302 responses to |
| 296 | POST requests must not be automatically redirected without |
| 297 | confirmation by the user. In reality, browsers do allow automatic |
| 298 | redirection of these responses, changing the POST to a GET, and |
| 299 | \module{urllib} reproduces this behaviour.} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 300 | |
| 301 | The parameters to the constructor are the same as those for |
| 302 | \class{URLopener}. |
Fred Drake | 47f11ce | 2001-04-12 20:26:49 +0000 | [diff] [blame] | 303 | |
Fred Drake | 0aa811c | 2001-10-20 04:24:09 +0000 | [diff] [blame] | 304 | \note{When performing basic authentication, a |
Fred Drake | 47f11ce | 2001-04-12 20:26:49 +0000 | [diff] [blame] | 305 | \class{FancyURLopener} instance calls its |
| 306 | \method{prompt_user_passwd()} method. The default implementation asks |
| 307 | the users for the required information on the controlling terminal. A |
| 308 | subclass may override this method to support more appropriate behavior |
Fred Drake | 0aa811c | 2001-10-20 04:24:09 +0000 | [diff] [blame] | 309 | if needed.} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 310 | \end{classdesc} |
| 311 | |
Georg Brandl | b925602 | 2005-08-24 18:46:39 +0000 | [diff] [blame] | 312 | \begin{excclassdesc}{ContentTooShortError}{msg\optional{, content}} |
| 313 | This exception is raised when the \function{urlretrieve()} function |
| 314 | detects that the amount of the downloaded data is less than the |
| 315 | expected amount (given by the \var{Content-Length} header). The |
| 316 | \member{content} attribute stores the downloaded (and supposedly |
| 317 | truncated) data. |
| 318 | \versionadded{2.5} |
| 319 | \end{excclassdesc} |
| 320 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 321 | Restrictions: |
| 322 | |
| 323 | \begin{itemize} |
| 324 | |
| 325 | \item |
| 326 | Currently, only the following protocols are supported: HTTP, (versions |
| 327 | 0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files. |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 328 | \indexii{HTTP}{protocol} |
| 329 | \indexii{Gopher}{protocol} |
| 330 | \indexii{FTP}{protocol} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 331 | |
| 332 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 333 | The caching feature of \function{urlretrieve()} has been disabled |
| 334 | until I find the time to hack proper processing of Expiration time |
| 335 | headers. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 336 | |
| 337 | \item |
Guido van Rossum | 6c4f003 | 1995-03-07 10:14:09 +0000 | [diff] [blame] | 338 | There should be a function to query whether a particular URL is in |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 339 | the cache. |
| 340 | |
| 341 | \item |
| 342 | For backward compatibility, if a URL appears to point to a local file |
| 343 | but the file can't be opened, the URL is re-interpreted using the FTP |
| 344 | protocol. This can sometimes cause confusing error messages. |
| 345 | |
| 346 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 347 | The \function{urlopen()} and \function{urlretrieve()} functions can |
| 348 | cause arbitrarily long delays while waiting for a network connection |
| 349 | to be set up. This means that it is difficult to build an interactive |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 350 | Web client using these functions without using threads. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 351 | |
| 352 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 353 | The data returned by \function{urlopen()} or \function{urlretrieve()} |
| 354 | is the raw data returned by the server. This may be binary data |
Georg Brandl | b925602 | 2005-08-24 18:46:39 +0000 | [diff] [blame] | 355 | (such as an image), plain text or (for example) HTML\index{HTML}. The |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 356 | HTTP\indexii{HTTP}{protocol} protocol provides type information in the |
| 357 | reply header, which can be inspected by looking at the |
Fred Drake | d86038d | 2001-08-03 18:39:36 +0000 | [diff] [blame] | 358 | \mailheader{Content-Type} header. For the |
| 359 | Gopher\indexii{Gopher}{protocol} protocol, type information is encoded |
| 360 | in the URL; there is currently no easy way to extract it. If the |
| 361 | returned data is HTML, you can use the module |
| 362 | \refmodule{htmllib}\refstmodindex{htmllib} to parse it. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 363 | |
| 364 | \item |
Andrew M. Kuchling | 2a510ce | 2004-07-13 14:03:31 +0000 | [diff] [blame] | 365 | The code handling the FTP\index{FTP} protocol cannot differentiate |
| 366 | between a file and a directory. This can lead to unexpected behavior |
| 367 | when attempting to read a URL that points to a file that is not |
| 368 | accessible. If the URL ends in a \code{/}, it is assumed to refer to |
| 369 | a directory and will be handled accordingly. But if an attempt to |
| 370 | read a file leads to a 550 error (meaning the URL cannot be found or |
| 371 | is not accessible, often for permission reasons), then the path is |
| 372 | treated as a directory in order to handle the case when a directory is |
| 373 | specified by a URL but the trailing \code{/} has been left off. This can |
| 374 | cause misleading results when you try to fetch a file whose read |
| 375 | permissions make it inaccessible; the FTP code will try to read it, |
| 376 | fail with a 550 error, and then perform a directory listing for the |
| 377 | unreadable file. If fine-grained control is needed, consider using the |
| 378 | \module{ftplib} module, subclassing \class{FancyURLOpener}, or changing |
| 379 | \var{_urlopener} to meet your needs. |
Brett Cannon | 71868e7 | 2004-07-13 00:48:42 +0000 | [diff] [blame] | 380 | |
| 381 | \item |
Fred Drake | 81c1735 | 2000-09-15 04:12:56 +0000 | [diff] [blame] | 382 | This module does not support the use of proxies which require |
| 383 | authentication. This may be implemented in the future. |
| 384 | |
| 385 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 386 | Although the \module{urllib} module contains (undocumented) routines |
| 387 | to parse and unparse URL strings, the recommended interface for URL |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 388 | manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 389 | |
| 390 | \end{itemize} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 391 | |
| 392 | |
| 393 | \subsection{URLopener Objects \label{urlopener-objs}} |
| 394 | \sectionauthor{Skip Montanaro}{skip@mojam.com} |
| 395 | |
| 396 | \class{URLopener} and \class{FancyURLopener} objects have the |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 397 | following attributes. |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 398 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 399 | \begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}} |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 400 | Open \var{fullurl} using the appropriate protocol. This method sets |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 401 | up cache and proxy information, then calls the appropriate open method with |
| 402 | its input arguments. If the scheme is not recognized, |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 403 | \method{open_unknown()} is called. The \var{data} argument |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 404 | has the same meaning as the \var{data} argument of \function{urlopen()}. |
| 405 | \end{methoddesc} |
| 406 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 407 | \begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 408 | Overridable interface to open unknown URL types. |
| 409 | \end{methoddesc} |
| 410 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 411 | \begin{methoddesc}[URLopener]{retrieve}{url\optional{, |
| 412 | filename\optional{, |
| 413 | reporthook\optional{, data}}}} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 414 | Retrieves the contents of \var{url} and places it in \var{filename}. The |
| 415 | return value is a tuple consisting of a local filename and either a |
| 416 | \class{mimetools.Message} object containing the response headers (for remote |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 417 | URLs) or \code{None} (for local URLs). The caller must then open and read the |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 418 | contents of \var{filename}. If \var{filename} is not given and the URL |
| 419 | refers to a local file, the input filename is returned. If the URL is |
| 420 | non-local and \var{filename} is not given, the filename is the output of |
| 421 | \function{tempfile.mktemp()} with a suffix that matches the suffix of the last |
| 422 | path component of the input URL. If \var{reporthook} is given, it must be |
| 423 | a function accepting three numeric parameters. It will be called after each |
| 424 | chunk of data is read from the network. \var{reporthook} is ignored for |
| 425 | local URLs. |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 426 | |
| 427 | If the \var{url} uses the \file{http:} scheme identifier, the optional |
| 428 | \var{data} argument may be given to specify a \code{POST} request |
| 429 | (normally the request type is \code{GET}). The \var{data} argument |
Fred Drake | 5100133 | 2000-12-15 23:57:51 +0000 | [diff] [blame] | 430 | must in standard \mimetype{application/x-www-form-urlencoded} format; |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 431 | see the \function{urlencode()} function below. |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 432 | \end{methoddesc} |
| 433 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 434 | \begin{memberdesc}[URLopener]{version} |
| 435 | Variable that specifies the user agent of the opener object. To get |
| 436 | \refmodule{urllib} to tell servers that it is a particular user agent, |
| 437 | set this in a subclass as a class variable or in the constructor |
| 438 | before calling the base constructor. |
| 439 | \end{memberdesc} |
| 440 | |
Fred Drake | 47f11ce | 2001-04-12 20:26:49 +0000 | [diff] [blame] | 441 | The \class{FancyURLopener} class offers one additional method that |
| 442 | should be overloaded to provide the appropriate behavior: |
| 443 | |
| 444 | \begin{methoddesc}[FancyURLopener]{prompt_user_passwd}{host, realm} |
| 445 | Return information needed to authenticate the user at the given host |
| 446 | in the specified security realm. The return value should be a tuple, |
| 447 | \code{(\var{user}, \var{password})}, which can be used for basic |
| 448 | authentication. |
| 449 | |
| 450 | The implementation prompts for this information on the terminal; an |
| 451 | application should override this method to use an appropriate |
| 452 | interaction model in the local environment. |
| 453 | \end{methoddesc} |
| 454 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 455 | |
| 456 | \subsection{Examples} |
| 457 | \nodename{Urllib Examples} |
| 458 | |
| 459 | Here is an example session that uses the \samp{GET} method to retrieve |
| 460 | a URL containing parameters: |
| 461 | |
| 462 | \begin{verbatim} |
| 463 | >>> import urllib |
| 464 | >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) |
| 465 | >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params) |
| 466 | >>> print f.read() |
| 467 | \end{verbatim} |
| 468 | |
| 469 | The following example uses the \samp{POST} method instead: |
| 470 | |
| 471 | \begin{verbatim} |
| 472 | >>> import urllib |
| 473 | >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) |
| 474 | >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params) |
| 475 | >>> print f.read() |
| 476 | \end{verbatim} |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 477 | |
| 478 | The following example uses an explicitly specified HTTP proxy, |
| 479 | overriding environment settings: |
| 480 | |
| 481 | \begin{verbatim} |
| 482 | >>> import urllib |
| 483 | >>> proxies = {'http': 'http://proxy.example.com:8080/'} |
| 484 | >>> opener = urllib.FancyURLopener(proxies) |
| 485 | >>> f = opener.open("http://www.python.org") |
| 486 | >>> f.read() |
| 487 | \end{verbatim} |
| 488 | |
| 489 | The following example uses no proxies at all, overriding environment |
| 490 | settings: |
| 491 | |
| 492 | \begin{verbatim} |
| 493 | >>> import urllib |
| 494 | >>> opener = urllib.FancyURLopener({}) |
| 495 | >>> f = opener.open("http://www.python.org/") |
| 496 | >>> f.read() |
| 497 | \end{verbatim} |