Fred Drake | 295da24 | 1998-08-10 19:42:37 +0000 | [diff] [blame] | 1 | \section{\module{urllib} --- |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 2 | Open arbitrary resources by URL} |
Fred Drake | b91e934 | 1998-07-23 17:59:49 +0000 | [diff] [blame] | 3 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 4 | \declaremodule{standard}{urllib} |
| 5 | \modulesynopsis{Open an arbitrary network resource by URL (requires sockets).} |
Fred Drake | b91e934 | 1998-07-23 17:59:49 +0000 | [diff] [blame] | 6 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 7 | \index{WWW} |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 8 | \index{World Wide Web} |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 9 | \index{URL} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 10 | |
Guido van Rossum | 8675115 | 1995-02-28 17:14:32 +0000 | [diff] [blame] | 11 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 12 | This module provides a high-level interface for fetching data across |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 13 | the World Wide Web. In particular, the \function{urlopen()} function |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 14 | is similar to the built-in function \function{open()}, but accepts |
| 15 | Universal Resource Locators (URLs) instead of filenames. Some |
| 16 | restrictions apply --- it can only open URLs for reading, and no seek |
| 17 | operations are available. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 18 | |
Fred Drake | f5eaa2e | 1997-12-15 22:13:50 +0000 | [diff] [blame] | 19 | It defines the following public functions: |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 20 | |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 21 | \begin{funcdesc}{urlopen}{url\optional{, data\optional{, proxies}}} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 22 | Open a network object denoted by a URL for reading. If the URL does |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 23 | not have a scheme identifier, or if it has \file{file:} as its scheme |
Brett Cannon | a2f8737 | 2003-04-29 04:11:12 +0000 | [diff] [blame] | 24 | identifier, this opens a local file (without universal newlines); |
| 25 | otherwise it opens a socket to a server somewhere on the network. If |
| 26 | the connection cannot be made, or if the server returns an error code, |
| 27 | the \exception{IOError} exception is raised. If all went well, a |
| 28 | file-like object is returned. This supports the following methods: |
| 29 | \method{read()}, \method{readline()}, \method{readlines()}, \method{fileno()}, |
| 30 | \method{close()}, \method{info()} and \method{geturl()}. It also has |
| 31 | proper support for the iterator protocol. |
Fred Drake | 504ca68 | 2004-03-25 16:51:12 +0000 | [diff] [blame^] | 32 | One caveat: the \method{read()} method, if the size argument is |
| 33 | omitted or negative, may not read until the end of the data stream; |
| 34 | there is no good way to determine that the entire stream from a socket |
| 35 | has been read in the general case. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 36 | |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 37 | Except for the \method{info()} and \method{geturl()} methods, |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 38 | these methods have the same interface as for |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 39 | file objects --- see section \ref{bltin-file-objects} in this |
| 40 | manual. (It is not a built-in file object, however, so it can't be |
Guido van Rossum | 470be14 | 1995-03-17 16:07:09 +0000 | [diff] [blame] | 41 | used at those few places where a true built-in file object is |
| 42 | required.) |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 43 | |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 44 | The \method{info()} method returns an instance of the class |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 45 | \class{mimetools.Message} containing meta-information associated |
| 46 | with the URL. When the method is HTTP, these headers are those |
| 47 | returned by the server at the head of the retrieved HTML page |
| 48 | (including Content-Length and Content-Type). When the method is FTP, |
| 49 | a Content-Length header will be present if (as is now usual) the |
| 50 | server passed back a file length in response to the FTP retrieval |
Guido van Rossum | 88e0b5b | 2001-08-23 13:38:15 +0000 | [diff] [blame] | 51 | request. A Content-Type header will be present if the MIME type can |
| 52 | be guessed. When the method is local-file, returned headers will include |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 53 | a Date representing the file's last-modified time, a Content-Length |
| 54 | giving file size, and a Content-Type containing a guess at the file's |
| 55 | type. See also the description of the |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 56 | \refmodule{mimetools}\refstmodindex{mimetools} module. |
| 57 | |
| 58 | The \method{geturl()} method returns the real URL of the page. In |
| 59 | some cases, the HTTP server redirects a client to another URL. The |
| 60 | \function{urlopen()} function handles this transparently, but in some |
| 61 | cases the caller needs to know which URL the client was redirected |
| 62 | to. The \method{geturl()} method can be used to get at this |
| 63 | redirected URL. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 64 | |
| 65 | If the \var{url} uses the \file{http:} scheme identifier, the optional |
| 66 | \var{data} argument may be given to specify a \code{POST} request |
| 67 | (normally the request type is \code{GET}). The \var{data} argument |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 68 | must be in standard \mimetype{application/x-www-form-urlencoded} format; |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 69 | see the \function{urlencode()} function below. |
| 70 | |
Fred Drake | aef0e89 | 2000-08-31 17:23:35 +0000 | [diff] [blame] | 71 | The \function{urlopen()} function works transparently with proxies |
Fred Drake | 81c1735 | 2000-09-15 04:12:56 +0000 | [diff] [blame] | 72 | which do not require authentication. In a \UNIX{} or Windows |
Fred Drake | aef0e89 | 2000-08-31 17:23:35 +0000 | [diff] [blame] | 73 | environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or |
| 74 | \envvar{gopher_proxy} environment variables to a URL that identifies |
| 75 | the proxy server before starting the Python interpreter. For example |
| 76 | (the \character{\%} is the command prompt): |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 77 | |
| 78 | \begin{verbatim} |
| 79 | % http_proxy="http://www.someproxy.com:3128" |
| 80 | % export http_proxy |
| 81 | % python |
| 82 | ... |
| 83 | \end{verbatim} |
| 84 | |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 85 | In a Windows environment, if no proxy environment variables are set, |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 86 | proxy settings are obtained from the registry's Internet Settings |
| 87 | section. |
| 88 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 89 | In a Macintosh environment, \function{urlopen()} will retrieve proxy |
| 90 | information from Internet\index{Internet Config} Config. |
| 91 | |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 92 | Alternatively, the optional \var{proxies} argument may be used to |
| 93 | explicitly specify proxies. It must be a dictionary mapping scheme |
| 94 | names to proxy URLs, where an empty dictionary causes no proxies to be |
| 95 | used, and \code{None} (the default value) causes environmental proxy |
| 96 | settings to be used as discussed above. For example: |
| 97 | |
| 98 | \begin{verbatim} |
| 99 | # Use http://www.someproxy.com:3128 for http proxying |
| 100 | proxies = proxies={'http': 'http://www.someproxy.com:3128'} |
| 101 | filehandle = urllib.urlopen(some_url, proxies=proxies) |
| 102 | # Don't use any proxies |
| 103 | filehandle = urllib.urlopen(some_url, proxies={}) |
| 104 | # Use proxies from environment - both versions are equivalent |
| 105 | filehandle = urllib.urlopen(some_url, proxies=None) |
| 106 | filehandle = urllib.urlopen(some_url) |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 107 | \end{verbatim} |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 108 | |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 109 | The \function{urlopen()} function does not support explicit proxy |
| 110 | specification. If you need to override environmental proxy settings, |
| 111 | use \class{URLopener}, or a subclass such as \class{FancyURLopener}. |
| 112 | |
Fred Drake | aef0e89 | 2000-08-31 17:23:35 +0000 | [diff] [blame] | 113 | Proxies which require authentication for use are not currently |
| 114 | supported; this is considered an implementation limitation. |
Fred Drake | a2c2595 | 2002-04-04 20:58:02 +0000 | [diff] [blame] | 115 | |
| 116 | \versionchanged[Added the \var{proxies} support]{2.3} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 117 | \end{funcdesc} |
| 118 | |
Fred Drake | 5100133 | 2000-12-15 23:57:51 +0000 | [diff] [blame] | 119 | \begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{, |
| 120 | reporthook\optional{, data}}}} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 121 | Copy a network object denoted by a URL to a local file, if necessary. |
Guido van Rossum | 6c4f003 | 1995-03-07 10:14:09 +0000 | [diff] [blame] | 122 | If the URL points to a local file, or a valid cached copy of the |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 123 | object exists, the object is not copied. Return a tuple |
| 124 | \code{(\var{filename}, \var{headers})} where \var{filename} is the |
| 125 | local file name under which the object can be found, and \var{headers} |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 126 | is whatever the \method{info()} method of the object returned by |
| 127 | \function{urlopen()} returned (for a remote object, possibly cached). |
| 128 | Exceptions are the same as for \function{urlopen()}. |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 129 | |
| 130 | The second argument, if present, specifies the file location to copy |
| 131 | to (if absent, the location will be a tempfile with a generated name). |
| 132 | The third argument, if present, is a hook function that will be called |
| 133 | once on establishment of the network connection and once after each |
| 134 | block read thereafter. The hook will be passed three arguments; a |
| 135 | count of blocks transferred so far, a block size in bytes, and the |
Fred Drake | 09b2957 | 1998-10-01 20:43:13 +0000 | [diff] [blame] | 136 | total size of the file. The third argument may be \code{-1} on older |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 137 | FTP servers which do not return a file size in response to a retrieval |
Guido van Rossum | 954b9ad | 1998-09-28 14:08:29 +0000 | [diff] [blame] | 138 | request. |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 139 | |
| 140 | If the \var{url} uses the \file{http:} scheme identifier, the optional |
| 141 | \var{data} argument may be given to specify a \code{POST} request |
| 142 | (normally the request type is \code{GET}). The \var{data} argument |
Fred Drake | 5100133 | 2000-12-15 23:57:51 +0000 | [diff] [blame] | 143 | must in standard \mimetype{application/x-www-form-urlencoded} format; |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 144 | see the \function{urlencode()} function below. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 145 | \end{funcdesc} |
| 146 | |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 147 | \begin{datadesc}{_urlopener} |
| 148 | The public functions \function{urlopen()} and |
| 149 | \function{urlretrieve()} create an instance of the |
| 150 | \class{FancyURLopener} class and use it to perform their requested |
| 151 | actions. To override this functionality, programmers can create a |
| 152 | subclass of \class{URLopener} or \class{FancyURLopener}, then assign |
Fred Drake | 4f4dbef | 2003-08-27 15:11:40 +0000 | [diff] [blame] | 153 | an instance of that class to the |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 154 | \code{urllib._urlopener} variable before calling the desired function. |
| 155 | For example, applications may want to specify a different |
| 156 | \mailheader{User-Agent} header than \class{URLopener} defines. This |
| 157 | can be accomplished with the following code: |
| 158 | |
| 159 | \begin{verbatim} |
| 160 | import urllib |
| 161 | |
| 162 | class AppURLopener(urllib.FancyURLopener): |
| 163 | def __init__(self, *args): |
| 164 | self.version = "App/1.7" |
| 165 | urllib.FancyURLopener.__init__(self, *args) |
| 166 | |
| 167 | urllib._urlopener = AppURLopener() |
| 168 | \end{verbatim} |
| 169 | \end{datadesc} |
| 170 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 171 | \begin{funcdesc}{urlcleanup}{} |
| 172 | Clear the cache that may have been built up by previous calls to |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 173 | \function{urlretrieve()}. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 174 | \end{funcdesc} |
| 175 | |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 176 | \begin{funcdesc}{quote}{string\optional{, safe}} |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 177 | Replace special characters in \var{string} using the \samp{\%xx} escape. |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 178 | Letters, digits, and the characters \character{_.-} are never quoted. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 179 | The optional \var{safe} parameter specifies additional characters |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 180 | that should not be quoted --- its default value is \code{'/'}. |
| 181 | |
Fred Drake | 10853c9 | 2000-07-28 13:51:27 +0000 | [diff] [blame] | 182 | Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}. |
Guido van Rossum | 8d40c84 | 1996-12-13 14:48:47 +0000 | [diff] [blame] | 183 | \end{funcdesc} |
| 184 | |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 185 | \begin{funcdesc}{quote_plus}{string\optional{, safe}} |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 186 | Like \function{quote()}, but also replaces spaces by plus signs, as |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 187 | required for quoting HTML form values. Plus signs in the original |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 188 | string are escaped unless they are included in \var{safe}. It also |
| 189 | does not have \var{safe} default to \code{'/'}. |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 190 | \end{funcdesc} |
| 191 | |
| 192 | \begin{funcdesc}{unquote}{string} |
Guido van Rossum | 6c4f003 | 1995-03-07 10:14:09 +0000 | [diff] [blame] | 193 | Replace \samp{\%xx} escapes by their single-character equivalent. |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 194 | |
Fred Drake | 10853c9 | 2000-07-28 13:51:27 +0000 | [diff] [blame] | 195 | Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}. |
Guido van Rossum | 61d34f4 | 1995-02-27 17:51:51 +0000 | [diff] [blame] | 196 | \end{funcdesc} |
| 197 | |
Guido van Rossum | 8d40c84 | 1996-12-13 14:48:47 +0000 | [diff] [blame] | 198 | \begin{funcdesc}{unquote_plus}{string} |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 199 | Like \function{unquote()}, but also replaces plus signs by spaces, as |
Guido van Rossum | 8d40c84 | 1996-12-13 14:48:47 +0000 | [diff] [blame] | 200 | required for unquoting HTML form values. |
| 201 | \end{funcdesc} |
| 202 | |
Skip Montanaro | 4fda21b | 2001-01-28 21:18:16 +0000 | [diff] [blame] | 203 | \begin{funcdesc}{urlencode}{query\optional{, doseq}} |
| 204 | Convert a mapping object or a sequence of two-element tuples to a |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 205 | ``url-encoded'' string, suitable to pass to |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 206 | \function{urlopen()} above as the optional \var{data} argument. This |
| 207 | is useful to pass a dictionary of form fields to a \code{POST} |
Fred Drake | 09b2957 | 1998-10-01 20:43:13 +0000 | [diff] [blame] | 208 | request. The resulting string is a series of |
| 209 | \code{\var{key}=\var{value}} pairs separated by \character{\&} |
| 210 | characters, where both \var{key} and \var{value} are quoted using |
Skip Montanaro | eda2844 | 2001-01-24 06:36:06 +0000 | [diff] [blame] | 211 | \function{quote_plus()} above. If the optional parameter \var{doseq} is |
| 212 | present and evaluates to true, individual \code{\var{key}=\var{value}} pairs |
| 213 | are generated for each element of the sequence. |
Skip Montanaro | 4fda21b | 2001-01-28 21:18:16 +0000 | [diff] [blame] | 214 | When a sequence of two-element tuples is used as the \var{query} argument, |
| 215 | the first element of each tuple is a key and the second is a value. The |
| 216 | order of parameters in the encoded string will match the order of parameter |
| 217 | tuples in the sequence. |
Fred Drake | d859d47 | 2003-04-24 16:22:47 +0000 | [diff] [blame] | 218 | The \refmodule{cgi} module provides the functions |
| 219 | \function{parse_qs()} and \function{parse_qsl()} which are used to |
| 220 | parse query strings into Python data structures. |
Guido van Rossum | 0af2f63 | 1998-07-22 21:34:21 +0000 | [diff] [blame] | 221 | \end{funcdesc} |
| 222 | |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 223 | \begin{funcdesc}{pathname2url}{path} |
| 224 | Convert the pathname \var{path} from the local syntax for a path to |
| 225 | the form used in the path component of a URL. This does not produce a |
| 226 | complete URL. The return value will already be quoted using the |
| 227 | \function{quote()} function. |
| 228 | \end{funcdesc} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 229 | |
Fred Drake | 88f015d | 2002-10-22 21:58:06 +0000 | [diff] [blame] | 230 | \begin{funcdesc}{url2pathname}{path} |
| 231 | Convert the path component \var{path} from an encoded URL to the local |
| 232 | syntax for a path. This does not accept a complete URL. This |
| 233 | function uses \function{unquote()} to decode \var{path}. |
| 234 | \end{funcdesc} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 235 | |
| 236 | \begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}} |
| 237 | Base class for opening and reading URLs. Unless you need to support |
| 238 | opening objects using schemes other than \file{http:}, \file{ftp:}, |
| 239 | \file{gopher:} or \file{file:}, you probably want to use |
| 240 | \class{FancyURLopener}. |
| 241 | |
| 242 | By default, the \class{URLopener} class sends a |
Fred Drake | d86038d | 2001-08-03 18:39:36 +0000 | [diff] [blame] | 243 | \mailheader{User-Agent} header of \samp{urllib/\var{VVV}}, where |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 244 | \var{VVV} is the \module{urllib} version number. Applications can |
Fred Drake | d86038d | 2001-08-03 18:39:36 +0000 | [diff] [blame] | 245 | define their own \mailheader{User-Agent} header by subclassing |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 246 | \class{URLopener} or \class{FancyURLopener} and setting the instance |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 247 | attribute \member{version} to an appropriate string value before the |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 248 | \method{open()} method is called. |
| 249 | |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 250 | The optional \var{proxies} parameter should be a dictionary mapping |
| 251 | scheme names to proxy URLs, where an empty dictionary turns proxies |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 252 | off completely. Its default value is \code{None}, in which case |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 253 | environmental proxy settings will be used if present, as discussed in |
| 254 | the definition of \function{urlopen()}, above. |
| 255 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 256 | Additional keyword parameters, collected in \var{x509}, are used for |
| 257 | authentication with the \file{https:} scheme. The keywords |
| 258 | \var{key_file} and \var{cert_file} are supported; both are needed to |
| 259 | actually retrieve a resource at an \file{https:} URL. |
| 260 | \end{classdesc} |
| 261 | |
| 262 | \begin{classdesc}{FancyURLopener}{...} |
| 263 | \class{FancyURLopener} subclasses \class{URLopener} providing default |
Martin v. Löwis | 162f081 | 2003-07-12 07:33:32 +0000 | [diff] [blame] | 264 | handling for the following HTTP response codes: 301, 302, 303, 307 and |
| 265 | 401. For the 30x response codes listed above, the |
| 266 | \mailheader{Location} header is used to fetch the actual URL. For 401 |
| 267 | response codes (authentication required), basic HTTP authentication is |
| 268 | performed. For the 30x response codes, recursion is bounded by the |
| 269 | value of the \var{maxtries} attribute, which defaults to 10. |
Raymond Hettinger | 024aaa1 | 2003-04-24 15:32:12 +0000 | [diff] [blame] | 270 | |
| 271 | \note{According to the letter of \rfc{2616}, 301 and 302 responses to |
| 272 | POST requests must not be automatically redirected without |
| 273 | confirmation by the user. In reality, browsers do allow automatic |
| 274 | redirection of these responses, changing the POST to a GET, and |
| 275 | \module{urllib} reproduces this behaviour.} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 276 | |
| 277 | The parameters to the constructor are the same as those for |
| 278 | \class{URLopener}. |
Fred Drake | 47f11ce | 2001-04-12 20:26:49 +0000 | [diff] [blame] | 279 | |
Fred Drake | 0aa811c | 2001-10-20 04:24:09 +0000 | [diff] [blame] | 280 | \note{When performing basic authentication, a |
Fred Drake | 47f11ce | 2001-04-12 20:26:49 +0000 | [diff] [blame] | 281 | \class{FancyURLopener} instance calls its |
| 282 | \method{prompt_user_passwd()} method. The default implementation asks |
| 283 | the users for the required information on the controlling terminal. A |
| 284 | subclass may override this method to support more appropriate behavior |
Fred Drake | 0aa811c | 2001-10-20 04:24:09 +0000 | [diff] [blame] | 285 | if needed.} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 286 | \end{classdesc} |
| 287 | |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 288 | Restrictions: |
| 289 | |
| 290 | \begin{itemize} |
| 291 | |
| 292 | \item |
| 293 | Currently, only the following protocols are supported: HTTP, (versions |
| 294 | 0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files. |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 295 | \indexii{HTTP}{protocol} |
| 296 | \indexii{Gopher}{protocol} |
| 297 | \indexii{FTP}{protocol} |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 298 | |
| 299 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 300 | The caching feature of \function{urlretrieve()} has been disabled |
| 301 | until I find the time to hack proper processing of Expiration time |
| 302 | headers. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 303 | |
| 304 | \item |
Guido van Rossum | 6c4f003 | 1995-03-07 10:14:09 +0000 | [diff] [blame] | 305 | There should be a function to query whether a particular URL is in |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 306 | the cache. |
| 307 | |
| 308 | \item |
| 309 | For backward compatibility, if a URL appears to point to a local file |
| 310 | but the file can't be opened, the URL is re-interpreted using the FTP |
| 311 | protocol. This can sometimes cause confusing error messages. |
| 312 | |
| 313 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 314 | The \function{urlopen()} and \function{urlretrieve()} functions can |
| 315 | cause arbitrarily long delays while waiting for a network connection |
| 316 | to be set up. This means that it is difficult to build an interactive |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 317 | Web client using these functions without using threads. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 318 | |
| 319 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 320 | The data returned by \function{urlopen()} or \function{urlretrieve()} |
| 321 | is the raw data returned by the server. This may be binary data |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 322 | (e.g. an image), plain text or (for example) HTML\index{HTML}. The |
| 323 | HTTP\indexii{HTTP}{protocol} protocol provides type information in the |
| 324 | reply header, which can be inspected by looking at the |
Fred Drake | d86038d | 2001-08-03 18:39:36 +0000 | [diff] [blame] | 325 | \mailheader{Content-Type} header. For the |
| 326 | Gopher\indexii{Gopher}{protocol} protocol, type information is encoded |
| 327 | in the URL; there is currently no easy way to extract it. If the |
| 328 | returned data is HTML, you can use the module |
| 329 | \refmodule{htmllib}\refstmodindex{htmllib} to parse it. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 330 | |
| 331 | \item |
Fred Drake | 81c1735 | 2000-09-15 04:12:56 +0000 | [diff] [blame] | 332 | This module does not support the use of proxies which require |
| 333 | authentication. This may be implemented in the future. |
| 334 | |
| 335 | \item |
Fred Drake | 6ef871c | 1998-03-12 06:52:05 +0000 | [diff] [blame] | 336 | Although the \module{urllib} module contains (undocumented) routines |
| 337 | to parse and unparse URL strings, the recommended interface for URL |
Fred Drake | 1ec71cb | 1999-02-22 22:42:14 +0000 | [diff] [blame] | 338 | manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}. |
Guido van Rossum | a8db1df | 1995-02-16 16:29:46 +0000 | [diff] [blame] | 339 | |
| 340 | \end{itemize} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 341 | |
| 342 | |
| 343 | \subsection{URLopener Objects \label{urlopener-objs}} |
| 344 | \sectionauthor{Skip Montanaro}{skip@mojam.com} |
| 345 | |
| 346 | \class{URLopener} and \class{FancyURLopener} objects have the |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 347 | following attributes. |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 348 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 349 | \begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}} |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 350 | Open \var{fullurl} using the appropriate protocol. This method sets |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 351 | up cache and proxy information, then calls the appropriate open method with |
| 352 | its input arguments. If the scheme is not recognized, |
Brett Cannon | 317ad7a | 2003-04-24 02:31:14 +0000 | [diff] [blame] | 353 | \method{open_unknown()} is called. The \var{data} argument |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 354 | has the same meaning as the \var{data} argument of \function{urlopen()}. |
| 355 | \end{methoddesc} |
| 356 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 357 | \begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 358 | Overridable interface to open unknown URL types. |
| 359 | \end{methoddesc} |
| 360 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 361 | \begin{methoddesc}[URLopener]{retrieve}{url\optional{, |
| 362 | filename\optional{, |
| 363 | reporthook\optional{, data}}}} |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 364 | Retrieves the contents of \var{url} and places it in \var{filename}. The |
| 365 | return value is a tuple consisting of a local filename and either a |
| 366 | \class{mimetools.Message} object containing the response headers (for remote |
Fred Drake | 5ca3a08 | 2002-04-04 20:34:36 +0000 | [diff] [blame] | 367 | URLs) or \code{None} (for local URLs). The caller must then open and read the |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 368 | contents of \var{filename}. If \var{filename} is not given and the URL |
| 369 | refers to a local file, the input filename is returned. If the URL is |
| 370 | non-local and \var{filename} is not given, the filename is the output of |
| 371 | \function{tempfile.mktemp()} with a suffix that matches the suffix of the last |
| 372 | path component of the input URL. If \var{reporthook} is given, it must be |
| 373 | a function accepting three numeric parameters. It will be called after each |
| 374 | chunk of data is read from the network. \var{reporthook} is ignored for |
| 375 | local URLs. |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 376 | |
| 377 | If the \var{url} uses the \file{http:} scheme identifier, the optional |
| 378 | \var{data} argument may be given to specify a \code{POST} request |
| 379 | (normally the request type is \code{GET}). The \var{data} argument |
Fred Drake | 5100133 | 2000-12-15 23:57:51 +0000 | [diff] [blame] | 380 | must in standard \mimetype{application/x-www-form-urlencoded} format; |
Fred Drake | 9fa4d61 | 2000-08-24 01:06:40 +0000 | [diff] [blame] | 381 | see the \function{urlencode()} function below. |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 382 | \end{methoddesc} |
| 383 | |
Fred Drake | dfca4dc | 2000-08-25 05:13:42 +0000 | [diff] [blame] | 384 | \begin{memberdesc}[URLopener]{version} |
| 385 | Variable that specifies the user agent of the opener object. To get |
| 386 | \refmodule{urllib} to tell servers that it is a particular user agent, |
| 387 | set this in a subclass as a class variable or in the constructor |
| 388 | before calling the base constructor. |
| 389 | \end{memberdesc} |
| 390 | |
Fred Drake | 47f11ce | 2001-04-12 20:26:49 +0000 | [diff] [blame] | 391 | The \class{FancyURLopener} class offers one additional method that |
| 392 | should be overloaded to provide the appropriate behavior: |
| 393 | |
| 394 | \begin{methoddesc}[FancyURLopener]{prompt_user_passwd}{host, realm} |
| 395 | Return information needed to authenticate the user at the given host |
| 396 | in the specified security realm. The return value should be a tuple, |
| 397 | \code{(\var{user}, \var{password})}, which can be used for basic |
| 398 | authentication. |
| 399 | |
| 400 | The implementation prompts for this information on the terminal; an |
| 401 | application should override this method to use an appropriate |
| 402 | interaction model in the local environment. |
| 403 | \end{methoddesc} |
| 404 | |
Fred Drake | 38e5d27 | 2000-04-03 20:13:55 +0000 | [diff] [blame] | 405 | |
| 406 | \subsection{Examples} |
| 407 | \nodename{Urllib Examples} |
| 408 | |
| 409 | Here is an example session that uses the \samp{GET} method to retrieve |
| 410 | a URL containing parameters: |
| 411 | |
| 412 | \begin{verbatim} |
| 413 | >>> import urllib |
| 414 | >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) |
| 415 | >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params) |
| 416 | >>> print f.read() |
| 417 | \end{verbatim} |
| 418 | |
| 419 | The following example uses the \samp{POST} method instead: |
| 420 | |
| 421 | \begin{verbatim} |
| 422 | >>> import urllib |
| 423 | >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) |
| 424 | >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params) |
| 425 | >>> print f.read() |
| 426 | \end{verbatim} |
Fred Drake | d216703 | 2002-04-04 20:09:50 +0000 | [diff] [blame] | 427 | |
| 428 | The following example uses an explicitly specified HTTP proxy, |
| 429 | overriding environment settings: |
| 430 | |
| 431 | \begin{verbatim} |
| 432 | >>> import urllib |
| 433 | >>> proxies = {'http': 'http://proxy.example.com:8080/'} |
| 434 | >>> opener = urllib.FancyURLopener(proxies) |
| 435 | >>> f = opener.open("http://www.python.org") |
| 436 | >>> f.read() |
| 437 | \end{verbatim} |
| 438 | |
| 439 | The following example uses no proxies at all, overriding environment |
| 440 | settings: |
| 441 | |
| 442 | \begin{verbatim} |
| 443 | >>> import urllib |
| 444 | >>> opener = urllib.FancyURLopener({}) |
| 445 | >>> f = opener.open("http://www.python.org/") |
| 446 | >>> f.read() |
| 447 | \end{verbatim} |