Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 1 | \section{\module{urllib2} --- |
| 2 | extensible library for opening URLs} |
| 3 | |
| 4 | \declaremodule{standard}{urllib2} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 5 | \moduleauthor{Jeremy Hylton}{jhylton@users.sourceforge.net} |
| 6 | \sectionauthor{Moshe Zadka}{moshez@users.sourceforge.net} |
| 7 | |
| 8 | \modulesynopsis{An extensible library for opening URLs using a variety of |
| 9 | protocols} |
| 10 | |
| 11 | The \module{urllib2} module defines functions and classes which help |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 12 | in opening URLs (mostly HTTP) in a complex world --- basic and digest |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 13 | authentication, redirections and more. |
| 14 | |
| 15 | The \module{urllib2} module defines the following functions: |
| 16 | |
| 17 | \begin{funcdesc}{urlopen}{url\optional{, data}} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 18 | Open the URL \var{url}, which can be either a string or a \class{Request} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 19 | object (currently the code checks that it really is a \class{Request} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 20 | instance, or an instance of a subclass of \class{Request}). |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 21 | |
| 22 | \var{data} should be a string, which specifies additional data to |
| 23 | send to the server. In HTTP requests, which are the only ones that |
| 24 | support \var{data}, it should be a buffer in the format of |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 25 | \mimetype{application/x-www-form-urlencoded}, for example one returned |
| 26 | from \function{urllib.urlencode()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 27 | |
| 28 | This function returns a file-like object with two additional methods: |
| 29 | |
| 30 | \begin{itemize} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 31 | \item \method{geturl()} --- return the URL of the resource retrieved |
| 32 | \item \method{info()} --- return the meta-information of the page, as |
| 33 | a dictionary-like object |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 34 | \end{itemize} |
| 35 | |
| 36 | Raises \exception{URLError} on errors. |
| 37 | \end{funcdesc} |
| 38 | |
| 39 | \begin{funcdesc}{install_opener}{opener} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 40 | Install an \class{OpenerDirector} instance as the default opener. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 41 | The code does not check for a real \class{OpenerDirector}, and any |
| 42 | class with the appropriate interface will work. |
| 43 | \end{funcdesc} |
| 44 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 45 | \begin{funcdesc}{build_opener}{\optional{handler, \moreargs}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 46 | Return an \class{OpenerDirector} instance, which chains the |
| 47 | handlers in the order given. \var{handler}s can be either instances |
| 48 | of \class{BaseHandler}, or subclasses of \class{BaseHandler} (in |
| 49 | which case it must be possible to call the constructor without |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 50 | any parameters). Instances of the following classes will be in |
| 51 | front of the \var{handler}s, unless the \var{handler}s contain |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 52 | them, instances of them or subclasses of them: |
| 53 | |
| 54 | \code{ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, |
| 55 | HTTPRedirectHandler, FTPHandler, FileHandler} |
| 56 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 57 | If the Python installation has SSL support (\function{socket.ssl()} |
| 58 | exists), \class{HTTPSHandler} will also be added. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 59 | \end{funcdesc} |
| 60 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 61 | |
| 62 | The following exceptions are raised as appropriate: |
| 63 | |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 64 | \begin{excdesc}{URLError} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 65 | The handlers raise this exception (or derived exceptions) when they |
| 66 | run into a problem. It is a subclass of \exception{IOError}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 67 | \end{excdesc} |
| 68 | |
| 69 | \begin{excdesc}{HTTPError} |
| 70 | A subclass of \exception{URLError}, it can also function as a |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 71 | non-exceptional file-like return value (the same thing that |
| 72 | \function{urlopen()} returns). This is useful when handling exotic |
| 73 | HTTP errors, such as requests for authentication. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 74 | \end{excdesc} |
| 75 | |
| 76 | \begin{excdesc}{GopherError} |
| 77 | A subclass of \exception{URLError}, this is the error raised by the |
| 78 | Gopher handler. |
| 79 | \end{excdesc} |
| 80 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 81 | |
| 82 | The following classes are provided: |
| 83 | |
| 84 | \begin{classdesc}{Request}{url\optional{, data\optional{, headers}}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 85 | This class is an abstraction of a URL request. |
| 86 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 87 | \var{url} should be a string which is a valid URL. For a description |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 88 | of \var{data} see the \method{add_data()} description. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 89 | \var{headers} should be a dictionary, and will be treated as if |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 90 | \method{add_header()} was called with each key and value as arguments. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 91 | \end{classdesc} |
| 92 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 93 | \begin{classdesc}{OpenerDirector}{} |
| 94 | The \class{OpenerDirector} class opens URLs via \class{BaseHandler}s |
| 95 | chained together. It manages the chaining of handlers, and recovery |
| 96 | from errors. |
| 97 | \end{classdesc} |
| 98 | |
| 99 | \begin{classdesc}{BaseHandler}{} |
| 100 | This is the base class for all registered handlers --- and handles only |
| 101 | the simple mechanics of registration. |
| 102 | \end{classdesc} |
| 103 | |
| 104 | \begin{classdesc}{HTTPDefaultErrorHandler}{} |
| 105 | A class which defines a default handler for HTTP error responses; all |
| 106 | responses are turned into \exception{HTTPError} exceptions. |
| 107 | \end{classdesc} |
| 108 | |
| 109 | \begin{classdesc}{HTTPRedirectHandler}{} |
| 110 | A class to handle redirections. |
| 111 | \end{classdesc} |
| 112 | |
| 113 | \begin{classdesc}{ProxyHandler}{\optional{proxies}} |
| 114 | Cause requests to go through a proxy. |
| 115 | If \var{proxies} is given, it must be a dictionary mapping |
| 116 | protocol names to URLs of proxies. |
| 117 | The default is to read the list of proxies from the environment |
Fred Drake | 4785246 | 2001-05-11 15:46:45 +0000 | [diff] [blame] | 118 | variables \var{protocol}_proxy. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 119 | \end{classdesc} |
| 120 | |
| 121 | \begin{classdesc}{HTTPPasswordMgr}{} |
| 122 | Keep a database of |
| 123 | \code{(\var{realm}, \var{uri}) -> (\var{user}, \var{password})} |
| 124 | mappings. |
| 125 | \end{classdesc} |
| 126 | |
| 127 | \begin{classdesc}{HTTPPasswordMgrWithDefaultRealm}{} |
| 128 | Keep a database of |
| 129 | \code{(\var{realm}, \var{uri}) -> (\var{user}, \var{password})} mappings. |
| 130 | A realm of \code{None} is considered a catch-all realm, which is searched |
| 131 | if no other realm fits. |
| 132 | \end{classdesc} |
| 133 | |
| 134 | \begin{classdesc}{AbstractBasicAuthHandler}{\optional{password_mgr}} |
| 135 | This is a mixin class that helps with HTTP authentication, both |
| 136 | to the remote host and to a proxy. |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 137 | \var{password_mgr}, if given, should be something that is compatible |
| 138 | with \class{HTTPPasswordMgr}; refer to section~\ref{http-password-mgr} |
| 139 | for information on the interface that must be supported. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 140 | \end{classdesc} |
| 141 | |
| 142 | \begin{classdesc}{HTTPBasicAuthHandler}{\optional{password_mgr}} |
| 143 | Handle authentication with the remote host. |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 144 | \var{password_mgr}, if given, should be something that is compatible |
| 145 | with \class{HTTPPasswordMgr}; refer to section~\ref{http-password-mgr} |
| 146 | for information on the interface that must be supported. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 147 | \end{classdesc} |
| 148 | |
| 149 | \begin{classdesc}{ProxyBasicAuthHandler}{\optional{password_mgr}} |
| 150 | Handle authentication with the proxy. |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 151 | \var{password_mgr}, if given, should be something that is compatible |
| 152 | with \class{HTTPPasswordMgr}; refer to section~\ref{http-password-mgr} |
| 153 | for information on the interface that must be supported. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 154 | \end{classdesc} |
| 155 | |
| 156 | \begin{classdesc}{AbstractDigestAuthHandler}{\optional{password_mgr}} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 157 | This is a mixin class that helps with HTTP authentication, both |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 158 | to the remote host and to a proxy. |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 159 | \var{password_mgr}, if given, should be something that is compatible |
| 160 | with \class{HTTPPasswordMgr}; refer to section~\ref{http-password-mgr} |
| 161 | for information on the interface that must be supported. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 162 | \end{classdesc} |
| 163 | |
| 164 | \begin{classdesc}{HTTPDigestAuthHandler}{\optional{password_mgr}} |
| 165 | Handle authentication with the remote host. |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 166 | \var{password_mgr}, if given, should be something that is compatible |
| 167 | with \class{HTTPPasswordMgr}; refer to section~\ref{http-password-mgr} |
| 168 | for information on the interface that must be supported. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 169 | \end{classdesc} |
| 170 | |
| 171 | \begin{classdesc}{ProxyDigestAuthHandler}{\optional{password_mgr}} |
| 172 | Handle authentication with the proxy. |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 173 | \var{password_mgr}, if given, should be something that is compatible |
| 174 | with \class{HTTPPasswordMgr}; refer to section~\ref{http-password-mgr} |
| 175 | for information on the interface that must be supported. |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 176 | \end{classdesc} |
| 177 | |
| 178 | \begin{classdesc}{HTTPHandler}{} |
| 179 | A class to handle opening of HTTP URLs. |
| 180 | \end{classdesc} |
| 181 | |
| 182 | \begin{classdesc}{HTTPSHandler}{} |
| 183 | A class to handle opening of HTTPS URLs. |
| 184 | \end{classdesc} |
| 185 | |
| 186 | \begin{classdesc}{FileHandler}{} |
| 187 | Open local files. |
| 188 | \end{classdesc} |
| 189 | |
| 190 | \begin{classdesc}{FTPHandler}{} |
| 191 | Open FTP URLs. |
| 192 | \end{classdesc} |
| 193 | |
| 194 | \begin{classdesc}{CacheFTPHandler}{} |
| 195 | Open FTP URLs, keeping a cache of open FTP connections to minimize |
| 196 | delays. |
| 197 | \end{classdesc} |
| 198 | |
| 199 | \begin{classdesc}{GopherHandler}{} |
| 200 | Open gopher URLs. |
| 201 | \end{classdesc} |
| 202 | |
| 203 | \begin{classdesc}{UnknownHandler}{} |
| 204 | A catch-all class to handle unknown URLs. |
| 205 | \end{classdesc} |
| 206 | |
| 207 | |
| 208 | \subsection{Request Objects \label{request-objects}} |
| 209 | |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 210 | The following methods describe all of \class{Request}'s public interface, |
| 211 | and so all must be overridden in subclasses. |
| 212 | |
| 213 | \begin{methoddesc}[Request]{add_data}{data} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 214 | Set the \class{Request} data to \var{data}. This is ignored |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 215 | by all handlers except HTTP handlers --- and there it should be an |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 216 | \mimetype{application/x-www-form-encoded} buffer, and will change the |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 217 | request to be \code{POST} rather than \code{GET}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 218 | \end{methoddesc} |
| 219 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 220 | \begin{methoddesc}[Request]{has_data}{} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 221 | Return whether the instance has a non-\code{None} data. |
| 222 | \end{methoddesc} |
| 223 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 224 | \begin{methoddesc}[Request]{get_data}{} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 225 | Return the instance's data. |
| 226 | \end{methoddesc} |
| 227 | |
| 228 | \begin{methoddesc}[Request]{add_header}{key, val} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 229 | Add another header to the request. Headers are currently ignored by |
| 230 | all handlers except HTTP handlers, where they are added to the list |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 231 | of headers sent to the server. Note that there cannot be more than |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 232 | one header with the same name, and later calls will overwrite |
| 233 | previous calls in case the \var{key} collides. Currently, this is |
| 234 | no loss of HTTP functionality, since all headers which have meaning |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 235 | when used more than once have a (header-specific) way of gaining the |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 236 | same functionality using only one header. |
| 237 | \end{methoddesc} |
| 238 | |
| 239 | \begin{methoddesc}[Request]{get_full_url}{} |
| 240 | Return the URL given in the constructor. |
| 241 | \end{methoddesc} |
| 242 | |
| 243 | \begin{methoddesc}[Request]{get_type}{} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 244 | Return the type of the URL --- also known as the scheme. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 245 | \end{methoddesc} |
| 246 | |
| 247 | \begin{methoddesc}[Request]{get_host}{} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 248 | Return the host to which a connection will be made. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 249 | \end{methoddesc} |
| 250 | |
| 251 | \begin{methoddesc}[Request]{get_selector}{} |
| 252 | Return the selector --- the part of the URL that is sent to |
| 253 | the server. |
| 254 | \end{methoddesc} |
| 255 | |
| 256 | \begin{methoddesc}[Request]{set_proxy}{host, type} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 257 | Prepare the request by connecting to a proxy server. The \var{host} |
| 258 | and \var{type} will replace those of the instance, and the instance's |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 259 | selector will be the original URL given in the constructor. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 260 | \end{methoddesc} |
| 261 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 262 | |
| 263 | \subsection{OpenerDirector Objects \label{opener-director-objects}} |
| 264 | |
| 265 | \class{OpenerDirector} instances have the following methods: |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 266 | |
| 267 | \begin{methoddesc}[OpenerDirector]{add_handler}{handler} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 268 | \var{handler} should be an instance of \class{BaseHandler}. The |
| 269 | following methods are searched, and added to the possible chains. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 270 | |
| 271 | \begin{itemize} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 272 | \item \method{\var{protocol}_open()} --- |
| 273 | signal that the handler knows how to open \var{protocol} URLs. |
| 274 | \item \method{\var{protocol}_error_\var{type}()} --- |
| 275 | signal that the handler knows how to handle \var{type} errors from |
| 276 | \var{protocol}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 277 | \end{itemize} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 278 | \end{methoddesc} |
| 279 | |
| 280 | \begin{methoddesc}[OpenerDirector]{close}{} |
| 281 | Explicitly break cycles, and delete all the handlers. |
| 282 | Because the \class{OpenerDirector} needs to know the registered handlers, |
| 283 | and a handler needs to know who the \class{OpenerDirector} who called |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 284 | it is, there is a reference cycle. Even though recent versions of Python |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 285 | have cycle-collection, it is sometimes preferable to explicitly break |
| 286 | the cycles. |
| 287 | \end{methoddesc} |
| 288 | |
| 289 | \begin{methoddesc}[OpenerDirector]{open}{url\optional{, data}} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 290 | Open the given \var{url} (which can be a request object or a string), |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 291 | optionally passing the given \var{data}. |
| 292 | Arguments, return values and exceptions raised are the same as those |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 293 | of \function{urlopen()} (which simply calls the \method{open()} method |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 294 | on the default installed \class{OpenerDirector}. |
| 295 | \end{methoddesc} |
| 296 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 297 | \begin{methoddesc}[OpenerDirector]{error}{proto\optional{, |
| 298 | arg\optional{, \moreargs}}} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 299 | Handle an error in a given protocol. This will call the registered |
| 300 | error handlers for the given protocol with the given arguments (which |
| 301 | are protocol specific). The HTTP protocol is a special case which |
| 302 | uses the HTTP response code to determine the specific error handler; |
| 303 | refer to the \method{http_error_*()} methods of the handler classes. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 304 | |
| 305 | Return values and exceptions raised are the same as those |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 306 | of \function{urlopen()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 307 | \end{methoddesc} |
| 308 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 309 | |
| 310 | \subsection{BaseHandler Objects \label{base-handler-objects}} |
| 311 | |
| 312 | \class{BaseHandler} objects provide a couple of methods that are |
| 313 | directly useful, and others that are meant to be used by derived |
| 314 | classes. These are intended for direct use: |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 315 | |
| 316 | \begin{methoddesc}[BaseHandler]{add_parent}{director} |
| 317 | Add a director as parent. |
| 318 | \end{methoddesc} |
| 319 | |
| 320 | \begin{methoddesc}[BaseHandler]{close}{} |
| 321 | Remove any parents. |
| 322 | \end{methoddesc} |
| 323 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 324 | The following members and methods should only be used by classes |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 325 | derived from \class{BaseHandler}: |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 326 | |
| 327 | \begin{memberdesc}[BaseHandler]{parent} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 328 | A valid \class{OpenerDirector}, which can be used to open using a |
| 329 | different protocol, or handle errors. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 330 | \end{memberdesc} |
| 331 | |
| 332 | \begin{methoddesc}[BaseHandler]{default_open}{req} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 333 | This method is \emph{not} defined in \class{BaseHandler}, but |
| 334 | subclasses should define it if they want to catch all URLs. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 335 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 336 | This method, if implemented, will be called by the parent |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 337 | \class{OpenerDirector}. It should return a file-like object as |
| 338 | described in the return value of the \method{open()} of |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 339 | \class{OpenerDirector}, or \code{None}. It should raise |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 340 | \exception{URLError}, unless a truly exceptional thing happens (for |
| 341 | example, \exception{MemoryError} should not be mapped to |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 342 | \exception{URLError}). |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 343 | |
| 344 | This method will be called before any protocol-specific open method. |
| 345 | \end{methoddesc} |
| 346 | |
Fred Drake | 4785246 | 2001-05-11 15:46:45 +0000 | [diff] [blame] | 347 | \begin{methoddescni}[BaseHandler]{\var{protocol}_open}{req} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 348 | This method is \emph{not} defined in \class{BaseHandler}, but |
| 349 | subclasses should define it if they want to handle URLs with the given |
| 350 | protocol. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 351 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 352 | This method, if defined, will be called by the parent |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 353 | \class{OpenerDirector}. Return values should be the same as for |
| 354 | \method{default_open()}. |
Fred Drake | 4785246 | 2001-05-11 15:46:45 +0000 | [diff] [blame] | 355 | \end{methoddescni} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 356 | |
| 357 | \begin{methoddesc}[BaseHandler]{unknown_open}{req} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 358 | This method is \var{not} defined in \class{BaseHandler}, but |
| 359 | subclasses should define it if they want to catch all URLs with no |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 360 | specific registered handler to open it. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 361 | |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 362 | This method, if implemented, will be called by the \member{parent} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 363 | \class{OpenerDirector}. Return values should be the same as for |
| 364 | \method{default_open()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 365 | \end{methoddesc} |
| 366 | |
| 367 | \begin{methoddesc}[BaseHandler]{http_error_default}{req, fp, code, msg, hdrs} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 368 | This method is \emph{not} defined in \class{BaseHandler}, but |
| 369 | subclasses should override it if they intend to provide a catch-all |
| 370 | for otherwise unhandled HTTP errors. It will be called automatically |
| 371 | by the \class{OpenerDirector} getting the error, and should not |
| 372 | normally be called in other circumstances. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 373 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 374 | \var{req} will be a \class{Request} object, \var{fp} will be a |
| 375 | file-like object with the HTTP error body, \var{code} will be the |
| 376 | three-digit code of the error, \var{msg} will be the user-visible |
| 377 | explanation of the code and \var{hdrs} will be a mapping object with |
| 378 | the headers of the error. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 379 | |
| 380 | Return values and exceptions raised should be the same as those |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 381 | of \function{urlopen()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 382 | \end{methoddesc} |
| 383 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 384 | \begin{methoddesc}[BaseHandler]{http_error_\var{nnn}}{req, fp, code, msg, hdrs} |
| 385 | \var{nnn} should be a three-digit HTTP error code. This method is |
| 386 | also not defined in \class{BaseHandler}, but will be called, if it |
| 387 | exists, on an instance of a subclass, when an HTTP error with code |
| 388 | \var{nnn} occurs. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 389 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 390 | Subclasses should override this method to handle specific HTTP |
| 391 | errors. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 392 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 393 | Arguments, return values and exceptions raised should be the same as |
| 394 | for \method{http_error_default()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 395 | \end{methoddesc} |
| 396 | |
| 397 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 398 | \subsection{HTTPRedirectHandler Objects \label{http-redirect-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 399 | |
Fred Drake | 0aa811c | 2001-10-20 04:24:09 +0000 | [diff] [blame] | 400 | \note{303 redirection is not supported by this version of |
| 401 | \module{urllib2}.} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 402 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 403 | \begin{methoddesc}[HTTPRedirectHandler]{http_error_301}{req, |
| 404 | fp, code, msg, hdrs} |
| 405 | Redirect to the \code{Location:} URL. This method is called by |
| 406 | the parent \class{OpenerDirector} when getting an HTTP |
| 407 | permanent-redirect response. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 408 | \end{methoddesc} |
| 409 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 410 | \begin{methoddesc}[HTTPRedirectHandler]{http_error_302}{req, |
| 411 | fp, code, msg, hdrs} |
| 412 | The same as \method{http_error_301()}, but called for the |
| 413 | temporary-redirect response. |
| 414 | \end{methoddesc} |
| 415 | |
| 416 | |
| 417 | \subsection{ProxyHandler Objects \label{proxy-handler}} |
| 418 | |
Fred Drake | 4785246 | 2001-05-11 15:46:45 +0000 | [diff] [blame] | 419 | \begin{methoddescni}[ProxyHandler]{\var{protocol}_open}{request} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 420 | The \class{ProxyHandler} will have a method |
| 421 | \method{\var{protocol}_open()} for every \var{protocol} which has a |
| 422 | proxy in the \var{proxies} dictionary given in the constructor. The |
| 423 | method will modify requests to go through the proxy, by calling |
| 424 | \code{request.set_proxy()}, and call the next handler in the chain to |
| 425 | actually execute the protocol. |
Fred Drake | 4785246 | 2001-05-11 15:46:45 +0000 | [diff] [blame] | 426 | \end{methoddescni} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 427 | |
| 428 | |
| 429 | \subsection{HTTPPasswordMgr Objects \label{http-password-mgr}} |
| 430 | |
| 431 | These methods are available on \class{HTTPPasswordMgr} and |
| 432 | \class{HTTPPasswordMgrWithDefaultRealm} objects. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 433 | |
| 434 | \begin{methoddesc}[HTTPPasswordMgr]{add_password}{realm, uri, user, passwd} |
| 435 | \var{uri} can be either a single URI, or a sequene of URIs. \var{realm}, |
| 436 | \var{user} and \var{passwd} must be strings. This causes |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 437 | \code{(\var{user}, \var{passwd})} to be used as authentication tokens |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 438 | when authentication for \var{realm} and a super-URI of any of the |
| 439 | given URIs is given. |
| 440 | \end{methoddesc} |
| 441 | |
| 442 | \begin{methoddesc}[HTTPPasswordMgr]{find_user_password}{realm, authuri} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 443 | Get user/password for given realm and URI, if any. This method will |
| 444 | return \code{(None, None)} if there is no matching user/password. |
| 445 | |
| 446 | For \class{HTTPPasswordMgrWithDefaultRealm} objects, the realm |
| 447 | \code{None} will be searched if the given \var{realm} has no matching |
| 448 | user/password. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 449 | \end{methoddesc} |
| 450 | |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 451 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 452 | \subsection{AbstractBasicAuthHandler Objects |
| 453 | \label{abstract-basic-auth-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 454 | |
| 455 | \begin{methoddesc}[AbstractBasicAuthHandler]{handle_authentication_request} |
| 456 | {authreq, host, req, headers} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 457 | Handle an authentication request by getting a user/password pair, and |
| 458 | re-trying the request. \var{authreq} should be the name of the header |
| 459 | where the information about the realm is included in the request, |
| 460 | \var{host} is the host to authenticate to, \var{req} should be the |
| 461 | (failed) \class{Request} object, and \var{headers} should be the error |
| 462 | headers. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 463 | \end{methoddesc} |
| 464 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 465 | |
| 466 | \subsection{HTTPBasicAuthHandler Objects |
| 467 | \label{http-basic-auth-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 468 | |
| 469 | \begin{methoddesc}[HTTPBasicAuthHandler]{http_error_401}{req, fp, code, |
| 470 | msg, hdrs} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 471 | Retry the request with authentication information, if available. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 472 | \end{methoddesc} |
| 473 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 474 | |
| 475 | \subsection{ProxyBasicAuthHandler Objects |
| 476 | \label{proxy-basic-auth-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 477 | |
| 478 | \begin{methoddesc}[ProxyBasicAuthHandler]{http_error_407}{req, fp, code, |
| 479 | msg, hdrs} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 480 | Retry the request with authentication information, if available. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 481 | \end{methoddesc} |
| 482 | |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 483 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 484 | \subsection{AbstractDigestAuthHandler Objects |
| 485 | \label{abstract-digest-auth-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 486 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 487 | \begin{methoddesc}[AbstractDigestAuthHandler]{handle_authentication_request} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 488 | {authreq, host, req, headers} |
| 489 | \var{authreq} should be the name of the header where the information about |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 490 | the realm is included in the request, \var{host} should be the host to |
| 491 | authenticate to, \var{req} should be the (failed) \class{Request} |
| 492 | object, and \var{headers} should be the error headers. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 493 | \end{methoddesc} |
| 494 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 495 | |
| 496 | \subsection{HTTPDigestAuthHandler Objects |
| 497 | \label{http-digest-auth-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 498 | |
| 499 | \begin{methoddesc}[HTTPDigestAuthHandler]{http_error_401}{req, fp, code, |
| 500 | msg, hdrs} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 501 | Retry the request with authentication information, if available. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 502 | \end{methoddesc} |
| 503 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 504 | |
| 505 | \subsection{ProxyDigestAuthHandler Objects |
| 506 | \label{proxy-digest-auth-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 507 | |
| 508 | \begin{methoddesc}[ProxyDigestAuthHandler]{http_error_407}{req, fp, code, |
| 509 | msg, hdrs} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 510 | Retry the request with authentication information, if available. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 511 | \end{methoddesc} |
| 512 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 513 | |
| 514 | \subsection{HTTPHandler Objects \label{http-handler-objects}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 515 | |
| 516 | \begin{methoddesc}[HTTPHandler]{http_open}{req} |
Fred Drake | 399bc8c | 2001-11-09 03:49:29 +0000 | [diff] [blame] | 517 | Send an HTTP request, which can be either GET or POST, depending on |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 518 | \code{\var{req}.has_data()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 519 | \end{methoddesc} |
| 520 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 521 | |
| 522 | \subsection{HTTPSHandler Objects \label{https-handler-objects}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 523 | |
| 524 | \begin{methoddesc}[HTTPSHandler]{https_open}{req} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 525 | Send an HTTPS request, which can be either GET or POST, depending on |
| 526 | \code{\var{req}.has_data()}. |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 527 | \end{methoddesc} |
| 528 | |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 529 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 530 | \subsection{FileHandler Objects \label{file-handler-objects}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 531 | |
| 532 | \begin{methoddesc}[FileHandler]{file_open}{req} |
| 533 | Open the file locally, if there is no host name, or |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 534 | the host name is \code{'localhost'}. Change the |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 535 | protocol to \code{ftp} otherwise, and retry opening |
| 536 | it using \member{parent}. |
| 537 | \end{methoddesc} |
| 538 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 539 | |
| 540 | \subsection{FTPHandler Objects \label{ftp-handler-objects}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 541 | |
| 542 | \begin{methoddesc}[FTPHandler]{ftp_open}{req} |
| 543 | Open the FTP file indicated by \var{req}. |
| 544 | The login is always done with empty username and password. |
| 545 | \end{methoddesc} |
| 546 | |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 547 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 548 | \subsection{CacheFTPHandler Objects \label{cacheftp-handler-objects}} |
| 549 | |
| 550 | \class{CacheFTPHandler} objects are \class{FTPHandler} objects with |
| 551 | the following additional methods: |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 552 | |
| 553 | \begin{methoddesc}[CacheFTPHandler]{setTimeout}{t} |
| 554 | Set timeout of connections to \var{t} seconds. |
| 555 | \end{methoddesc} |
| 556 | |
| 557 | \begin{methoddesc}[CacheFTPHandler]{setMaxConns}{m} |
| 558 | Set maximum number of cached connections to \var{m}. |
| 559 | \end{methoddesc} |
| 560 | |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 561 | |
| 562 | \subsection{GopherHandler Objects \label{gopher-handler}} |
Moshe Zadka | 8a18e99 | 2001-03-01 08:40:42 +0000 | [diff] [blame] | 563 | |
| 564 | \begin{methoddesc}[GopherHandler]{gopher_open}{req} |
| 565 | Open the gopher resource indicated by \var{req}. |
| 566 | \end{methoddesc} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 567 | |
| 568 | |
| 569 | \subsection{UnknownHandler Objects \label{unknown-handler-objects}} |
| 570 | |
Fred Drake | a939911 | 2001-07-05 21:14:03 +0000 | [diff] [blame] | 571 | \begin{methoddesc}[UnknownHandler]{unknown_open}{} |
Fred Drake | 93c8671 | 2001-03-02 20:39:34 +0000 | [diff] [blame] | 572 | Raise a \exception{URLError} exception. |
| 573 | \end{methoddesc} |