Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 1 | :mod:`http.cookiejar` --- Cookie handling for HTTP clients |
| 2 | ========================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 3 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 4 | .. module:: http.cookiejar |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 5 | :synopsis: Classes for automatic handling of HTTP cookies. |
| 6 | .. moduleauthor:: John J. Lee <jjl@pobox.com> |
| 7 | .. sectionauthor:: John J. Lee <jjl@pobox.com> |
| 8 | |
Raymond Hettinger | 469271d | 2011-01-27 20:38:46 +0000 | [diff] [blame] | 9 | **Source code:** :source:`Lib/http/cookiejar.py` |
| 10 | |
| 11 | -------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 12 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 13 | The :mod:`http.cookiejar` module defines classes for automatic handling of HTTP |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 14 | cookies. It is useful for accessing web sites that require small pieces of data |
| 15 | -- :dfn:`cookies` -- to be set on the client machine by an HTTP response from a |
| 16 | web server, and then returned to the server in later HTTP requests. |
| 17 | |
| 18 | Both the regular Netscape cookie protocol and the protocol defined by |
| 19 | :rfc:`2965` are handled. RFC 2965 handling is switched off by default. |
| 20 | :rfc:`2109` cookies are parsed as Netscape cookies and subsequently treated |
| 21 | either as Netscape or RFC 2965 cookies according to the 'policy' in effect. |
| 22 | Note that the great majority of cookies on the Internet are Netscape cookies. |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 23 | :mod:`http.cookiejar` attempts to follow the de-facto Netscape cookie protocol (which |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 24 | differs substantially from that set out in the original Netscape specification), |
| 25 | including taking note of the ``max-age`` and ``port`` cookie-attributes |
| 26 | introduced with RFC 2965. |
| 27 | |
| 28 | .. note:: |
| 29 | |
| 30 | The various named parameters found in :mailheader:`Set-Cookie` and |
| 31 | :mailheader:`Set-Cookie2` headers (eg. ``domain`` and ``expires``) are |
| 32 | conventionally referred to as :dfn:`attributes`. To distinguish them from |
| 33 | Python attributes, the documentation for this module uses the term |
| 34 | :dfn:`cookie-attribute` instead. |
| 35 | |
| 36 | |
| 37 | The module defines the following exception: |
| 38 | |
| 39 | |
| 40 | .. exception:: LoadError |
| 41 | |
| 42 | Instances of :class:`FileCookieJar` raise this exception on failure to load |
Antoine Pitrou | 62ab10a0 | 2011-10-12 20:10:51 +0200 | [diff] [blame] | 43 | cookies from a file. :exc:`LoadError` is a subclass of :exc:`OSError`. |
| 44 | |
| 45 | .. versionchanged:: 3.3 |
| 46 | LoadError was made a subclass of :exc:`OSError` instead of |
| 47 | :exc:`IOError`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 48 | |
| 49 | |
| 50 | The following classes are provided: |
| 51 | |
| 52 | |
| 53 | .. class:: CookieJar(policy=None) |
| 54 | |
| 55 | *policy* is an object implementing the :class:`CookiePolicy` interface. |
| 56 | |
| 57 | The :class:`CookieJar` class stores HTTP cookies. It extracts cookies from HTTP |
| 58 | requests, and returns them in HTTP responses. :class:`CookieJar` instances |
| 59 | automatically expire contained cookies when necessary. Subclasses are also |
| 60 | responsible for storing and retrieving cookies from a file or database. |
| 61 | |
| 62 | |
| 63 | .. class:: FileCookieJar(filename, delayload=None, policy=None) |
| 64 | |
| 65 | *policy* is an object implementing the :class:`CookiePolicy` interface. For the |
| 66 | other arguments, see the documentation for the corresponding attributes. |
| 67 | |
| 68 | A :class:`CookieJar` which can load cookies from, and perhaps save cookies to, a |
| 69 | file on disk. Cookies are **NOT** loaded from the named file until either the |
| 70 | :meth:`load` or :meth:`revert` method is called. Subclasses of this class are |
| 71 | documented in section :ref:`file-cookie-jar-classes`. |
| 72 | |
| 73 | |
| 74 | .. class:: CookiePolicy() |
| 75 | |
| 76 | This class is responsible for deciding whether each cookie should be accepted |
| 77 | from / returned to the server. |
| 78 | |
| 79 | |
| 80 | .. class:: DefaultCookiePolicy( blocked_domains=None, allowed_domains=None, netscape=True, rfc2965=False, rfc2109_as_netscape=None, hide_cookie2=False, strict_domain=False, strict_rfc2965_unverifiable=True, strict_ns_unverifiable=False, strict_ns_domain=DefaultCookiePolicy.DomainLiberal, strict_ns_set_initial_dollar=False, strict_ns_set_path=False ) |
| 81 | |
| 82 | Constructor arguments should be passed as keyword arguments only. |
| 83 | *blocked_domains* is a sequence of domain names that we never accept cookies |
| 84 | from, nor return cookies to. *allowed_domains* if not :const:`None`, this is a |
| 85 | sequence of the only domains for which we accept and return cookies. For all |
| 86 | other arguments, see the documentation for :class:`CookiePolicy` and |
| 87 | :class:`DefaultCookiePolicy` objects. |
| 88 | |
| 89 | :class:`DefaultCookiePolicy` implements the standard accept / reject rules for |
| 90 | Netscape and RFC 2965 cookies. By default, RFC 2109 cookies (ie. cookies |
| 91 | received in a :mailheader:`Set-Cookie` header with a version cookie-attribute of |
| 92 | 1) are treated according to the RFC 2965 rules. However, if RFC 2965 handling |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 93 | is turned off or :attr:`rfc2109_as_netscape` is ``True``, RFC 2109 cookies are |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 94 | 'downgraded' by the :class:`CookieJar` instance to Netscape cookies, by |
| 95 | setting the :attr:`version` attribute of the :class:`Cookie` instance to 0. |
| 96 | :class:`DefaultCookiePolicy` also provides some parameters to allow some |
| 97 | fine-tuning of policy. |
| 98 | |
| 99 | |
| 100 | .. class:: Cookie() |
| 101 | |
| 102 | This class represents Netscape, RFC 2109 and RFC 2965 cookies. It is not |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 103 | expected that users of :mod:`http.cookiejar` construct their own :class:`Cookie` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 104 | instances. Instead, if necessary, call :meth:`make_cookies` on a |
| 105 | :class:`CookieJar` instance. |
| 106 | |
| 107 | |
| 108 | .. seealso:: |
| 109 | |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 110 | Module :mod:`urllib.request` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 111 | URL opening with automatic cookie handling. |
| 112 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 113 | Module :mod:`http.cookies` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 114 | HTTP cookie classes, principally useful for server-side code. The |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 115 | :mod:`http.cookiejar` and :mod:`http.cookies` modules do not depend on each |
| 116 | other. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 117 | |
Christian Heimes | dd15f6c | 2008-03-16 00:07:10 +0000 | [diff] [blame] | 118 | http://wp.netscape.com/newsref/std/cookie_spec.html |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 119 | The specification of the original Netscape cookie protocol. Though this is |
| 120 | still the dominant protocol, the 'Netscape cookie protocol' implemented by all |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 121 | the major browsers (and :mod:`http.cookiejar`) only bears a passing resemblance to |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 122 | the one sketched out in ``cookie_spec.html``. |
| 123 | |
| 124 | :rfc:`2109` - HTTP State Management Mechanism |
| 125 | Obsoleted by RFC 2965. Uses :mailheader:`Set-Cookie` with version=1. |
| 126 | |
| 127 | :rfc:`2965` - HTTP State Management Mechanism |
| 128 | The Netscape protocol with the bugs fixed. Uses :mailheader:`Set-Cookie2` in |
| 129 | place of :mailheader:`Set-Cookie`. Not widely used. |
| 130 | |
| 131 | http://kristol.org/cookie/errata.html |
| 132 | Unfinished errata to RFC 2965. |
| 133 | |
| 134 | :rfc:`2964` - Use of HTTP State Management |
| 135 | |
| 136 | .. _cookie-jar-objects: |
| 137 | |
| 138 | CookieJar and FileCookieJar Objects |
| 139 | ----------------------------------- |
| 140 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 141 | :class:`CookieJar` objects support the :term:`iterator` protocol for iterating over |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 142 | contained :class:`Cookie` objects. |
| 143 | |
| 144 | :class:`CookieJar` has the following methods: |
| 145 | |
| 146 | |
| 147 | .. method:: CookieJar.add_cookie_header(request) |
| 148 | |
| 149 | Add correct :mailheader:`Cookie` header to *request*. |
| 150 | |
| 151 | If policy allows (ie. the :attr:`rfc2965` and :attr:`hide_cookie2` attributes of |
| 152 | the :class:`CookieJar`'s :class:`CookiePolicy` instance are true and false |
| 153 | respectively), the :mailheader:`Cookie2` header is also added when appropriate. |
| 154 | |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 155 | The *request* object (usually a :class:`urllib.request..Request` instance) |
| 156 | must support the methods :meth:`get_full_url`, :meth:`get_host`, |
Senthil Kumaran | 05ec6ac | 2013-05-23 05:27:38 -0700 | [diff] [blame] | 157 | :meth:`get_type`, :meth:`unverifiable`, :meth:`has_header`, |
| 158 | :meth:`get_header`, :meth:`header_items`, :meth:`add_unredirected_header` |
| 159 | and :attr:`origin_req_host` attribute as documented by |
| 160 | :mod:`urllib.request`. |
| 161 | |
| 162 | .. versionchanged:: 3.3 |
| 163 | |
| 164 | *request* object needs :attr:`origin_req_host` attribute. Dependency on a |
| 165 | deprecated method :meth:`get_origin_req_host` has been removed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 166 | |
| 167 | |
| 168 | .. method:: CookieJar.extract_cookies(response, request) |
| 169 | |
| 170 | Extract cookies from HTTP *response* and store them in the :class:`CookieJar`, |
| 171 | where allowed by policy. |
| 172 | |
| 173 | The :class:`CookieJar` will look for allowable :mailheader:`Set-Cookie` and |
| 174 | :mailheader:`Set-Cookie2` headers in the *response* argument, and store cookies |
| 175 | as appropriate (subject to the :meth:`CookiePolicy.set_ok` method's approval). |
| 176 | |
Georg Brandl | 83e9f4c | 2008-06-12 18:52:31 +0000 | [diff] [blame] | 177 | The *response* object (usually the result of a call to |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 178 | :meth:`urllib.request.urlopen`, or similar) should support an :meth:`info` |
| 179 | method, which returns a :class:`email.message.Message` instance. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 180 | |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 181 | The *request* object (usually a :class:`urllib.request.Request` instance) |
| 182 | must support the methods :meth:`get_full_url`, :meth:`get_host`, |
Senthil Kumaran | 05ec6ac | 2013-05-23 05:27:38 -0700 | [diff] [blame] | 183 | :meth:`unverifiable`, and :attr:`origin_req_host` attribute, as documented |
| 184 | by :mod:`urllib.request`. The request is used to set default values for |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 185 | cookie-attributes as well as for checking that the cookie is allowed to be |
| 186 | set. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 187 | |
Senthil Kumaran | 05ec6ac | 2013-05-23 05:27:38 -0700 | [diff] [blame] | 188 | .. versionchanged:: 3.3 |
| 189 | |
| 190 | *request* object needs :attr:`origin_req_host` attribute. Dependency on a |
| 191 | deprecated method :meth:`get_origin_req_host` has been removed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 192 | |
| 193 | .. method:: CookieJar.set_policy(policy) |
| 194 | |
| 195 | Set the :class:`CookiePolicy` instance to be used. |
| 196 | |
| 197 | |
| 198 | .. method:: CookieJar.make_cookies(response, request) |
| 199 | |
| 200 | Return sequence of :class:`Cookie` objects extracted from *response* object. |
| 201 | |
| 202 | See the documentation for :meth:`extract_cookies` for the interfaces required of |
| 203 | the *response* and *request* arguments. |
| 204 | |
| 205 | |
| 206 | .. method:: CookieJar.set_cookie_if_ok(cookie, request) |
| 207 | |
| 208 | Set a :class:`Cookie` if policy says it's OK to do so. |
| 209 | |
| 210 | |
| 211 | .. method:: CookieJar.set_cookie(cookie) |
| 212 | |
| 213 | Set a :class:`Cookie`, without checking with policy to see whether or not it |
| 214 | should be set. |
| 215 | |
| 216 | |
| 217 | .. method:: CookieJar.clear([domain[, path[, name]]]) |
| 218 | |
| 219 | Clear some cookies. |
| 220 | |
| 221 | If invoked without arguments, clear all cookies. If given a single argument, |
| 222 | only cookies belonging to that *domain* will be removed. If given two arguments, |
| 223 | cookies belonging to the specified *domain* and URL *path* are removed. If |
| 224 | given three arguments, then the cookie with the specified *domain*, *path* and |
| 225 | *name* is removed. |
| 226 | |
| 227 | Raises :exc:`KeyError` if no matching cookie exists. |
| 228 | |
| 229 | |
| 230 | .. method:: CookieJar.clear_session_cookies() |
| 231 | |
| 232 | Discard all session cookies. |
| 233 | |
| 234 | Discards all contained cookies that have a true :attr:`discard` attribute |
| 235 | (usually because they had either no ``max-age`` or ``expires`` cookie-attribute, |
| 236 | or an explicit ``discard`` cookie-attribute). For interactive browsers, the end |
| 237 | of a session usually corresponds to closing the browser window. |
| 238 | |
| 239 | Note that the :meth:`save` method won't save session cookies anyway, unless you |
| 240 | ask otherwise by passing a true *ignore_discard* argument. |
| 241 | |
| 242 | :class:`FileCookieJar` implements the following additional methods: |
| 243 | |
| 244 | |
| 245 | .. method:: FileCookieJar.save(filename=None, ignore_discard=False, ignore_expires=False) |
| 246 | |
| 247 | Save cookies to a file. |
| 248 | |
| 249 | This base class raises :exc:`NotImplementedError`. Subclasses may leave this |
| 250 | method unimplemented. |
| 251 | |
| 252 | *filename* is the name of file in which to save cookies. If *filename* is not |
| 253 | specified, :attr:`self.filename` is used (whose default is the value passed to |
| 254 | the constructor, if any); if :attr:`self.filename` is :const:`None`, |
| 255 | :exc:`ValueError` is raised. |
| 256 | |
| 257 | *ignore_discard*: save even cookies set to be discarded. *ignore_expires*: save |
| 258 | even cookies that have expired |
| 259 | |
| 260 | The file is overwritten if it already exists, thus wiping all the cookies it |
| 261 | contains. Saved cookies can be restored later using the :meth:`load` or |
| 262 | :meth:`revert` methods. |
| 263 | |
| 264 | |
| 265 | .. method:: FileCookieJar.load(filename=None, ignore_discard=False, ignore_expires=False) |
| 266 | |
| 267 | Load cookies from a file. |
| 268 | |
| 269 | Old cookies are kept unless overwritten by newly loaded ones. |
| 270 | |
| 271 | Arguments are as for :meth:`save`. |
| 272 | |
| 273 | The named file must be in the format understood by the class, or |
Antoine Pitrou | 62ab10a0 | 2011-10-12 20:10:51 +0200 | [diff] [blame] | 274 | :exc:`LoadError` will be raised. Also, :exc:`OSError` may be raised, for |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 275 | example if the file does not exist. |
| 276 | |
Antoine Pitrou | 62ab10a0 | 2011-10-12 20:10:51 +0200 | [diff] [blame] | 277 | .. versionchanged:: 3.3 |
| 278 | :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. |
| 279 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 280 | |
| 281 | .. method:: FileCookieJar.revert(filename=None, ignore_discard=False, ignore_expires=False) |
| 282 | |
| 283 | Clear all cookies and reload cookies from a saved file. |
| 284 | |
| 285 | :meth:`revert` can raise the same exceptions as :meth:`load`. If there is a |
| 286 | failure, the object's state will not be altered. |
| 287 | |
| 288 | :class:`FileCookieJar` instances have the following public attributes: |
| 289 | |
| 290 | |
| 291 | .. attribute:: FileCookieJar.filename |
| 292 | |
| 293 | Filename of default file in which to keep cookies. This attribute may be |
| 294 | assigned to. |
| 295 | |
| 296 | |
| 297 | .. attribute:: FileCookieJar.delayload |
| 298 | |
| 299 | If true, load cookies lazily from disk. This attribute should not be assigned |
| 300 | to. This is only a hint, since this only affects performance, not behaviour |
| 301 | (unless the cookies on disk are changing). A :class:`CookieJar` object may |
| 302 | ignore it. None of the :class:`FileCookieJar` classes included in the standard |
| 303 | library lazily loads cookies. |
| 304 | |
| 305 | |
| 306 | .. _file-cookie-jar-classes: |
| 307 | |
| 308 | FileCookieJar subclasses and co-operation with web browsers |
| 309 | ----------------------------------------------------------- |
| 310 | |
Senthil Kumaran | aba088e | 2010-07-11 05:01:52 +0000 | [diff] [blame] | 311 | The following :class:`CookieJar` subclasses are provided for reading and |
Serhiy Storchaka | a4d170d | 2013-12-23 18:20:51 +0200 | [diff] [blame] | 312 | writing. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 313 | |
| 314 | .. class:: MozillaCookieJar(filename, delayload=None, policy=None) |
| 315 | |
| 316 | A :class:`FileCookieJar` that can load from and save cookies to disk in the |
| 317 | Mozilla ``cookies.txt`` file format (which is also used by the Lynx and Netscape |
| 318 | browsers). |
| 319 | |
| 320 | .. note:: |
| 321 | |
| 322 | This loses information about RFC 2965 cookies, and also about newer or |
| 323 | non-standard cookie-attributes such as ``port``. |
| 324 | |
| 325 | .. warning:: |
| 326 | |
| 327 | Back up your cookies before saving if you have cookies whose loss / corruption |
| 328 | would be inconvenient (there are some subtleties which may lead to slight |
| 329 | changes in the file over a load / save round-trip). |
| 330 | |
| 331 | Also note that cookies saved while Mozilla is running will get clobbered by |
| 332 | Mozilla. |
| 333 | |
| 334 | |
| 335 | .. class:: LWPCookieJar(filename, delayload=None, policy=None) |
| 336 | |
| 337 | A :class:`FileCookieJar` that can load from and save cookies to disk in format |
| 338 | compatible with the libwww-perl library's ``Set-Cookie3`` file format. This is |
| 339 | convenient if you want to store cookies in a human-readable file. |
| 340 | |
| 341 | |
| 342 | .. _cookie-policy-objects: |
| 343 | |
| 344 | CookiePolicy Objects |
| 345 | -------------------- |
| 346 | |
| 347 | Objects implementing the :class:`CookiePolicy` interface have the following |
| 348 | methods: |
| 349 | |
| 350 | |
| 351 | .. method:: CookiePolicy.set_ok(cookie, request) |
| 352 | |
| 353 | Return boolean value indicating whether cookie should be accepted from server. |
| 354 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 355 | *cookie* is a :class:`Cookie` instance. *request* is an object |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 356 | implementing the interface defined by the documentation for |
| 357 | :meth:`CookieJar.extract_cookies`. |
| 358 | |
| 359 | |
| 360 | .. method:: CookiePolicy.return_ok(cookie, request) |
| 361 | |
| 362 | Return boolean value indicating whether cookie should be returned to server. |
| 363 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 364 | *cookie* is a :class:`Cookie` instance. *request* is an object |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 365 | implementing the interface defined by the documentation for |
| 366 | :meth:`CookieJar.add_cookie_header`. |
| 367 | |
| 368 | |
| 369 | .. method:: CookiePolicy.domain_return_ok(domain, request) |
| 370 | |
| 371 | Return false if cookies should not be returned, given cookie domain. |
| 372 | |
| 373 | This method is an optimization. It removes the need for checking every cookie |
| 374 | with a particular domain (which might involve reading many files). Returning |
| 375 | true from :meth:`domain_return_ok` and :meth:`path_return_ok` leaves all the |
| 376 | work to :meth:`return_ok`. |
| 377 | |
| 378 | If :meth:`domain_return_ok` returns true for the cookie domain, |
| 379 | :meth:`path_return_ok` is called for the cookie path. Otherwise, |
| 380 | :meth:`path_return_ok` and :meth:`return_ok` are never called for that cookie |
| 381 | domain. If :meth:`path_return_ok` returns true, :meth:`return_ok` is called |
| 382 | with the :class:`Cookie` object itself for a full check. Otherwise, |
| 383 | :meth:`return_ok` is never called for that cookie path. |
| 384 | |
| 385 | Note that :meth:`domain_return_ok` is called for every *cookie* domain, not just |
| 386 | for the *request* domain. For example, the function might be called with both |
| 387 | ``".example.com"`` and ``"www.example.com"`` if the request domain is |
| 388 | ``"www.example.com"``. The same goes for :meth:`path_return_ok`. |
| 389 | |
| 390 | The *request* argument is as documented for :meth:`return_ok`. |
| 391 | |
| 392 | |
| 393 | .. method:: CookiePolicy.path_return_ok(path, request) |
| 394 | |
| 395 | Return false if cookies should not be returned, given cookie path. |
| 396 | |
| 397 | See the documentation for :meth:`domain_return_ok`. |
| 398 | |
| 399 | In addition to implementing the methods above, implementations of the |
| 400 | :class:`CookiePolicy` interface must also supply the following attributes, |
| 401 | indicating which protocols should be used, and how. All of these attributes may |
| 402 | be assigned to. |
| 403 | |
| 404 | |
| 405 | .. attribute:: CookiePolicy.netscape |
| 406 | |
| 407 | Implement Netscape protocol. |
| 408 | |
| 409 | |
| 410 | .. attribute:: CookiePolicy.rfc2965 |
| 411 | |
| 412 | Implement RFC 2965 protocol. |
| 413 | |
| 414 | |
| 415 | .. attribute:: CookiePolicy.hide_cookie2 |
| 416 | |
| 417 | Don't add :mailheader:`Cookie2` header to requests (the presence of this header |
| 418 | indicates to the server that we understand RFC 2965 cookies). |
| 419 | |
| 420 | The most useful way to define a :class:`CookiePolicy` class is by subclassing |
| 421 | from :class:`DefaultCookiePolicy` and overriding some or all of the methods |
| 422 | above. :class:`CookiePolicy` itself may be used as a 'null policy' to allow |
| 423 | setting and receiving any and all cookies (this is unlikely to be useful). |
| 424 | |
| 425 | |
| 426 | .. _default-cookie-policy-objects: |
| 427 | |
| 428 | DefaultCookiePolicy Objects |
| 429 | --------------------------- |
| 430 | |
| 431 | Implements the standard rules for accepting and returning cookies. |
| 432 | |
| 433 | Both RFC 2965 and Netscape cookies are covered. RFC 2965 handling is switched |
| 434 | off by default. |
| 435 | |
| 436 | The easiest way to provide your own policy is to override this class and call |
| 437 | its methods in your overridden implementations before adding your own additional |
| 438 | checks:: |
| 439 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 440 | import http.cookiejar |
| 441 | class MyCookiePolicy(http.cookiejar.DefaultCookiePolicy): |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 442 | def set_ok(self, cookie, request): |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 443 | if not http.cookiejar.DefaultCookiePolicy.set_ok(self, cookie, request): |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 444 | return False |
| 445 | if i_dont_want_to_store_this_cookie(cookie): |
| 446 | return False |
| 447 | return True |
| 448 | |
| 449 | In addition to the features required to implement the :class:`CookiePolicy` |
| 450 | interface, this class allows you to block and allow domains from setting and |
| 451 | receiving cookies. There are also some strictness switches that allow you to |
| 452 | tighten up the rather loose Netscape protocol rules a little bit (at the cost of |
| 453 | blocking some benign cookies). |
| 454 | |
| 455 | A domain blacklist and whitelist is provided (both off by default). Only domains |
| 456 | not in the blacklist and present in the whitelist (if the whitelist is active) |
| 457 | participate in cookie setting and returning. Use the *blocked_domains* |
| 458 | constructor argument, and :meth:`blocked_domains` and |
| 459 | :meth:`set_blocked_domains` methods (and the corresponding argument and methods |
| 460 | for *allowed_domains*). If you set a whitelist, you can turn it off again by |
| 461 | setting it to :const:`None`. |
| 462 | |
| 463 | Domains in block or allow lists that do not start with a dot must equal the |
| 464 | cookie domain to be matched. For example, ``"example.com"`` matches a blacklist |
| 465 | entry of ``"example.com"``, but ``"www.example.com"`` does not. Domains that do |
| 466 | start with a dot are matched by more specific domains too. For example, both |
| 467 | ``"www.example.com"`` and ``"www.coyote.example.com"`` match ``".example.com"`` |
| 468 | (but ``"example.com"`` itself does not). IP addresses are an exception, and |
| 469 | must match exactly. For example, if blocked_domains contains ``"192.168.1.2"`` |
| 470 | and ``".168.1.2"``, 192.168.1.2 is blocked, but 193.168.1.2 is not. |
| 471 | |
| 472 | :class:`DefaultCookiePolicy` implements the following additional methods: |
| 473 | |
| 474 | |
| 475 | .. method:: DefaultCookiePolicy.blocked_domains() |
| 476 | |
| 477 | Return the sequence of blocked domains (as a tuple). |
| 478 | |
| 479 | |
| 480 | .. method:: DefaultCookiePolicy.set_blocked_domains(blocked_domains) |
| 481 | |
| 482 | Set the sequence of blocked domains. |
| 483 | |
| 484 | |
| 485 | .. method:: DefaultCookiePolicy.is_blocked(domain) |
| 486 | |
| 487 | Return whether *domain* is on the blacklist for setting or receiving cookies. |
| 488 | |
| 489 | |
| 490 | .. method:: DefaultCookiePolicy.allowed_domains() |
| 491 | |
| 492 | Return :const:`None`, or the sequence of allowed domains (as a tuple). |
| 493 | |
| 494 | |
| 495 | .. method:: DefaultCookiePolicy.set_allowed_domains(allowed_domains) |
| 496 | |
| 497 | Set the sequence of allowed domains, or :const:`None`. |
| 498 | |
| 499 | |
| 500 | .. method:: DefaultCookiePolicy.is_not_allowed(domain) |
| 501 | |
| 502 | Return whether *domain* is not on the whitelist for setting or receiving |
| 503 | cookies. |
| 504 | |
| 505 | :class:`DefaultCookiePolicy` instances have the following attributes, which are |
| 506 | all initialised from the constructor arguments of the same name, and which may |
| 507 | all be assigned to. |
| 508 | |
| 509 | |
| 510 | .. attribute:: DefaultCookiePolicy.rfc2109_as_netscape |
| 511 | |
| 512 | If true, request that the :class:`CookieJar` instance downgrade RFC 2109 cookies |
| 513 | (ie. cookies received in a :mailheader:`Set-Cookie` header with a version |
| 514 | cookie-attribute of 1) to Netscape cookies by setting the version attribute of |
| 515 | the :class:`Cookie` instance to 0. The default value is :const:`None`, in which |
| 516 | case RFC 2109 cookies are downgraded if and only if RFC 2965 handling is turned |
| 517 | off. Therefore, RFC 2109 cookies are downgraded by default. |
| 518 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 519 | |
| 520 | General strictness switches: |
| 521 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 522 | .. attribute:: DefaultCookiePolicy.strict_domain |
| 523 | |
| 524 | Don't allow sites to set two-component domains with country-code top-level |
| 525 | domains like ``.co.uk``, ``.gov.uk``, ``.co.nz``.etc. This is far from perfect |
| 526 | and isn't guaranteed to work! |
| 527 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 528 | |
Georg Brandl | 55ac8f0 | 2007-09-01 13:51:09 +0000 | [diff] [blame] | 529 | RFC 2965 protocol strictness switches: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 530 | |
| 531 | .. attribute:: DefaultCookiePolicy.strict_rfc2965_unverifiable |
| 532 | |
| 533 | Follow RFC 2965 rules on unverifiable transactions (usually, an unverifiable |
| 534 | transaction is one resulting from a redirect or a request for an image hosted on |
| 535 | another site). If this is false, cookies are *never* blocked on the basis of |
| 536 | verifiability |
| 537 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 538 | |
Georg Brandl | 55ac8f0 | 2007-09-01 13:51:09 +0000 | [diff] [blame] | 539 | Netscape protocol strictness switches: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 540 | |
| 541 | .. attribute:: DefaultCookiePolicy.strict_ns_unverifiable |
| 542 | |
| 543 | apply RFC 2965 rules on unverifiable transactions even to Netscape cookies |
| 544 | |
| 545 | |
| 546 | .. attribute:: DefaultCookiePolicy.strict_ns_domain |
| 547 | |
| 548 | Flags indicating how strict to be with domain-matching rules for Netscape |
| 549 | cookies. See below for acceptable values. |
| 550 | |
| 551 | |
| 552 | .. attribute:: DefaultCookiePolicy.strict_ns_set_initial_dollar |
| 553 | |
| 554 | Ignore cookies in Set-Cookie: headers that have names starting with ``'$'``. |
| 555 | |
| 556 | |
| 557 | .. attribute:: DefaultCookiePolicy.strict_ns_set_path |
| 558 | |
| 559 | Don't allow setting cookies whose path doesn't path-match request URI. |
| 560 | |
| 561 | :attr:`strict_ns_domain` is a collection of flags. Its value is constructed by |
| 562 | or-ing together (for example, ``DomainStrictNoDots|DomainStrictNonDomain`` means |
| 563 | both flags are set). |
| 564 | |
| 565 | |
| 566 | .. attribute:: DefaultCookiePolicy.DomainStrictNoDots |
| 567 | |
| 568 | When setting cookies, the 'host prefix' must not contain a dot (eg. |
| 569 | ``www.foo.bar.com`` can't set a cookie for ``.bar.com``, because ``www.foo`` |
| 570 | contains a dot). |
| 571 | |
| 572 | |
| 573 | .. attribute:: DefaultCookiePolicy.DomainStrictNonDomain |
| 574 | |
| 575 | Cookies that did not explicitly specify a ``domain`` cookie-attribute can only |
| 576 | be returned to a domain equal to the domain that set the cookie (eg. |
| 577 | ``spam.example.com`` won't be returned cookies from ``example.com`` that had no |
| 578 | ``domain`` cookie-attribute). |
| 579 | |
| 580 | |
| 581 | .. attribute:: DefaultCookiePolicy.DomainRFC2965Match |
| 582 | |
| 583 | When setting cookies, require a full RFC 2965 domain-match. |
| 584 | |
| 585 | The following attributes are provided for convenience, and are the most useful |
| 586 | combinations of the above flags: |
| 587 | |
| 588 | |
| 589 | .. attribute:: DefaultCookiePolicy.DomainLiberal |
| 590 | |
| 591 | Equivalent to 0 (ie. all of the above Netscape domain strictness flags switched |
| 592 | off). |
| 593 | |
| 594 | |
| 595 | .. attribute:: DefaultCookiePolicy.DomainStrict |
| 596 | |
| 597 | Equivalent to ``DomainStrictNoDots|DomainStrictNonDomain``. |
| 598 | |
| 599 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 600 | Cookie Objects |
| 601 | -------------- |
| 602 | |
| 603 | :class:`Cookie` instances have Python attributes roughly corresponding to the |
| 604 | standard cookie-attributes specified in the various cookie standards. The |
| 605 | correspondence is not one-to-one, because there are complicated rules for |
| 606 | assigning default values, because the ``max-age`` and ``expires`` |
| 607 | cookie-attributes contain equivalent information, and because RFC 2109 cookies |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 608 | may be 'downgraded' by :mod:`http.cookiejar` from version 1 to version 0 (Netscape) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 609 | cookies. |
| 610 | |
| 611 | Assignment to these attributes should not be necessary other than in rare |
| 612 | circumstances in a :class:`CookiePolicy` method. The class does not enforce |
| 613 | internal consistency, so you should know what you're doing if you do that. |
| 614 | |
| 615 | |
| 616 | .. attribute:: Cookie.version |
| 617 | |
| 618 | Integer or :const:`None`. Netscape cookies have :attr:`version` 0. RFC 2965 and |
| 619 | RFC 2109 cookies have a ``version`` cookie-attribute of 1. However, note that |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 620 | :mod:`http.cookiejar` may 'downgrade' RFC 2109 cookies to Netscape cookies, in which |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 621 | case :attr:`version` is 0. |
| 622 | |
| 623 | |
| 624 | .. attribute:: Cookie.name |
| 625 | |
| 626 | Cookie name (a string). |
| 627 | |
| 628 | |
| 629 | .. attribute:: Cookie.value |
| 630 | |
| 631 | Cookie value (a string), or :const:`None`. |
| 632 | |
| 633 | |
| 634 | .. attribute:: Cookie.port |
| 635 | |
| 636 | String representing a port or a set of ports (eg. '80', or '80,8080'), or |
| 637 | :const:`None`. |
| 638 | |
| 639 | |
| 640 | .. attribute:: Cookie.path |
| 641 | |
| 642 | Cookie path (a string, eg. ``'/acme/rocket_launchers'``). |
| 643 | |
| 644 | |
| 645 | .. attribute:: Cookie.secure |
| 646 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 647 | ``True`` if cookie should only be returned over a secure connection. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 648 | |
| 649 | |
| 650 | .. attribute:: Cookie.expires |
| 651 | |
| 652 | Integer expiry date in seconds since epoch, or :const:`None`. See also the |
| 653 | :meth:`is_expired` method. |
| 654 | |
| 655 | |
| 656 | .. attribute:: Cookie.discard |
| 657 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 658 | ``True`` if this is a session cookie. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 659 | |
| 660 | |
| 661 | .. attribute:: Cookie.comment |
| 662 | |
| 663 | String comment from the server explaining the function of this cookie, or |
| 664 | :const:`None`. |
| 665 | |
| 666 | |
| 667 | .. attribute:: Cookie.comment_url |
| 668 | |
| 669 | URL linking to a comment from the server explaining the function of this cookie, |
| 670 | or :const:`None`. |
| 671 | |
| 672 | |
| 673 | .. attribute:: Cookie.rfc2109 |
| 674 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 675 | ``True`` if this cookie was received as an RFC 2109 cookie (ie. the cookie |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 676 | arrived in a :mailheader:`Set-Cookie` header, and the value of the Version |
| 677 | cookie-attribute in that header was 1). This attribute is provided because |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 678 | :mod:`http.cookiejar` may 'downgrade' RFC 2109 cookies to Netscape cookies, in |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 679 | which case :attr:`version` is 0. |
| 680 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 681 | |
| 682 | .. attribute:: Cookie.port_specified |
| 683 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 684 | ``True`` if a port or set of ports was explicitly specified by the server (in the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 685 | :mailheader:`Set-Cookie` / :mailheader:`Set-Cookie2` header). |
| 686 | |
| 687 | |
| 688 | .. attribute:: Cookie.domain_specified |
| 689 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 690 | ``True`` if a domain was explicitly specified by the server. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 691 | |
| 692 | |
| 693 | .. attribute:: Cookie.domain_initial_dot |
| 694 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 695 | ``True`` if the domain explicitly specified by the server began with a dot |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 696 | (``'.'``). |
| 697 | |
| 698 | Cookies may have additional non-standard cookie-attributes. These may be |
| 699 | accessed using the following methods: |
| 700 | |
| 701 | |
| 702 | .. method:: Cookie.has_nonstandard_attr(name) |
| 703 | |
| 704 | Return true if cookie has the named cookie-attribute. |
| 705 | |
| 706 | |
| 707 | .. method:: Cookie.get_nonstandard_attr(name, default=None) |
| 708 | |
| 709 | If cookie has the named cookie-attribute, return its value. Otherwise, return |
| 710 | *default*. |
| 711 | |
| 712 | |
| 713 | .. method:: Cookie.set_nonstandard_attr(name, value) |
| 714 | |
| 715 | Set the value of the named cookie-attribute. |
| 716 | |
| 717 | The :class:`Cookie` class also defines the following method: |
| 718 | |
| 719 | |
Hynek Schlawack | dfa4652 | 2012-05-21 11:01:54 +0200 | [diff] [blame] | 720 | .. method:: Cookie.is_expired(now=None) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 721 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 722 | ``True`` if cookie has passed the time at which the server requested it should |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 723 | expire. If *now* is given (in seconds since the epoch), return whether the |
| 724 | cookie has expired at the specified time. |
| 725 | |
| 726 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 727 | Examples |
| 728 | -------- |
| 729 | |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 730 | The first example shows the most common usage of :mod:`http.cookiejar`:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 731 | |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 732 | import http.cookiejar, urllib.request |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 733 | cj = http.cookiejar.CookieJar() |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 734 | opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 735 | r = opener.open("http://example.com/") |
| 736 | |
| 737 | This example illustrates how to open a URL using your Netscape, Mozilla, or Lynx |
| 738 | cookies (assumes Unix/Netscape convention for location of the cookies file):: |
| 739 | |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 740 | import os, http.cookiejar, urllib.request |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 741 | cj = http.cookiejar.MozillaCookieJar() |
Éric Araujo | 4dcf502 | 2011-03-25 20:31:50 +0100 | [diff] [blame] | 742 | cj.load(os.path.join(os.path.expanduser("~"), ".netscape", "cookies.txt")) |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 743 | opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 744 | r = opener.open("http://example.com/") |
| 745 | |
| 746 | The next example illustrates the use of :class:`DefaultCookiePolicy`. Turn on |
| 747 | RFC 2965 cookies, be more strict about domains when setting and returning |
| 748 | Netscape cookies, and block some domains from setting cookies or having them |
| 749 | returned:: |
| 750 | |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 751 | import urllib.request |
Georg Brandl | 2442015 | 2008-05-26 16:32:26 +0000 | [diff] [blame] | 752 | from http.cookiejar import CookieJar, DefaultCookiePolicy |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 753 | policy = DefaultCookiePolicy( |
| 754 | rfc2965=True, strict_ns_domain=Policy.DomainStrict, |
| 755 | blocked_domains=["ads.net", ".ads.net"]) |
| 756 | cj = CookieJar(policy) |
Georg Brandl | 029986a | 2008-06-23 11:44:14 +0000 | [diff] [blame] | 757 | opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 758 | r = opener.open("http://example.com/") |
| 759 | |