| \section{\module{urlparse} --- |
| Parse URLs into components} |
| \declaremodule{standard}{urlparse} |
| |
| \modulesynopsis{Parse URLs into components.} |
| |
| \index{WWW} |
| \index{World Wide Web} |
| \index{URL} |
| \indexii{URL}{parsing} |
| \indexii{relative}{URL} |
| |
| |
| This module defines a standard interface to break Uniform Resource |
| Locator (URL) strings up in components (addressing scheme, network |
| location, path etc.), to combine the components back into a URL |
| string, and to convert a ``relative URL'' to an absolute URL given a |
| ``base URL.'' |
| |
| The module has been designed to match the Internet RFC on Relative |
| Uniform Resource Locators (and discovered a bug in an earlier |
| draft!). It supports the following URL schemes: |
| \code{file}, \code{ftp}, \code{gopher}, \code{hdl}, \code{http}, |
| \code{https}, \code{imap}, \code{mailto}, \code{mms}, \code{news}, |
| \code{nntp}, \code{prospero}, \code{rsync}, \code{rtsp}, \code{rtspu}, |
| \code{sftp}, \code{shttp}, \code{sip}, \code{sips}, \code{snews}, \code{svn}, |
| \code{svn+ssh}, \code{telnet}, \code{wais}. |
| |
| \versionadded[Support for the \code{sftp} and \code{sips} schemes]{2.5} |
| |
| The \module{urlparse} module defines the following functions: |
| |
| \begin{funcdesc}{urlparse}{urlstring\optional{, |
| default_scheme\optional{, allow_fragments}}} |
| Parse a URL into six components, returning a 6-tuple. This |
| corresponds to the general structure of a URL: |
| \code{\var{scheme}://\var{netloc}/\var{path};\var{parameters}?\var{query}\#\var{fragment}}. |
| Each tuple item is a string, possibly empty. |
| The components are not broken up in smaller parts (for example, the network |
| location is a single string), and \% escapes are not expanded. |
| The delimiters as shown above are not part of the result, |
| except for a leading slash in the \var{path} component, which is |
| retained if present. For example: |
| |
| \begin{verbatim} |
| >>> from urlparse import urlparse |
| >>> o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html') |
| >>> o |
| ('http', 'www.cwi.nl:80', '/%7Eguido/Python.html', '', '', '') |
| >>> o.scheme |
| 'http' |
| >>> o.port |
| 80 |
| >>> o.geturl() |
| 'http://www.cwi.nl:80/%7Eguido/Python.html' |
| \end{verbatim} |
| |
| If the \var{default_scheme} argument is specified, it gives the |
| default addressing scheme, to be used only if the URL does not |
| specify one. The default value for this argument is the empty string. |
| |
| If the \var{allow_fragments} argument is false, fragment identifiers |
| are not allowed, even if the URL's addressing scheme normally does |
| support them. The default value for this argument is \constant{True}. |
| |
| The return value is actually an instance of a subclass of |
| \pytype{tuple}. This class has the following additional read-only |
| convenience attributes: |
| |
| \begin{tableiv}{l|c|l|c}{member}{Attribute}{Index}{Value}{Value if not present} |
| \lineiv{scheme} {0} {URL scheme specifier} {empty string} |
| \lineiv{netloc} {1} {Network location part} {empty string} |
| \lineiv{path} {2} {Hierarchical path} {empty string} |
| \lineiv{params} {3} {Parameters for last path element} {empty string} |
| \lineiv{query} {4} {Query component} {empty string} |
| \lineiv{fragment}{5} {Fragment identifier} {empty string} |
| \lineiv{username}{ } {User name} {\constant{None}} |
| \lineiv{password}{ } {Password} {\constant{None}} |
| \lineiv{hostname}{ } {Host name (lower case)} {\constant{None}} |
| \lineiv{port} { } {Port number as integer, if present} {\constant{None}} |
| \end{tableiv} |
| |
| See section~\ref{urlparse-result-object}, ``Results of |
| \function{urlparse()} and \function{urlsplit()},'' for more |
| information on the result object. |
| |
| \versionchanged[Added attributes to return value]{2.5} |
| \end{funcdesc} |
| |
| \begin{funcdesc}{urlunparse}{parts} |
| Construct a URL from a tuple as returned by \code{urlparse()}. |
| The \var{parts} argument can be any six-item iterable. |
| This may result in a slightly different, but equivalent URL, if the |
| URL that was parsed originally had unnecessary delimiters (for example, |
| a ? with an empty query; the RFC states that these are equivalent). |
| \end{funcdesc} |
| |
| \begin{funcdesc}{urlsplit}{urlstring\optional{, |
| default_scheme\optional{, allow_fragments}}} |
| This is similar to \function{urlparse()}, but does not split the |
| params from the URL. This should generally be used instead of |
| \function{urlparse()} if the more recent URL syntax allowing |
| parameters to be applied to each segment of the \var{path} portion of |
| the URL (see \rfc{2396}) is wanted. A separate function is needed to |
| separate the path segments and parameters. This function returns a |
| 5-tuple: (addressing scheme, network location, path, query, fragment |
| identifier). |
| |
| The return value is actually an instance of a subclass of |
| \pytype{tuple}. This class has the following additional read-only |
| convenience attributes: |
| |
| \begin{tableiv}{l|c|l|c}{member}{Attribute}{Index}{Value}{Value if not present} |
| \lineiv{scheme} {0} {URL scheme specifier} {empty string} |
| \lineiv{netloc} {1} {Network location part} {empty string} |
| \lineiv{path} {2} {Hierarchical path} {empty string} |
| \lineiv{query} {3} {Query component} {empty string} |
| \lineiv{fragment} {4} {Fragment identifier} {empty string} |
| \lineiv{username} { } {User name} {\constant{None}} |
| \lineiv{password} { } {Password} {\constant{None}} |
| \lineiv{hostname} { } {Host name (lower case)} {\constant{None}} |
| \lineiv{port} { } {Port number as integer, if present} {\constant{None}} |
| \end{tableiv} |
| |
| See section~\ref{urlparse-result-object}, ``Results of |
| \function{urlparse()} and \function{urlsplit()},'' for more |
| information on the result object. |
| |
| \versionadded{2.2} |
| \versionchanged[Added attributes to return value]{2.5} |
| \end{funcdesc} |
| |
| \begin{funcdesc}{urlunsplit}{parts} |
| Combine the elements of a tuple as returned by \function{urlsplit()} |
| into a complete URL as a string. |
| The \var{parts} argument can be any five-item iterable. |
| This may result in a slightly different, but equivalent URL, if the |
| URL that was parsed originally had unnecessary delimiters (for example, |
| a ? with an empty query; the RFC states that these are equivalent). |
| \versionadded{2.2} |
| \end{funcdesc} |
| |
| \begin{funcdesc}{urljoin}{base, url\optional{, allow_fragments}} |
| Construct a full (``absolute'') URL by combining a ``base URL'' |
| (\var{base}) with another URL (\var{url}). Informally, this |
| uses components of the base URL, in particular the addressing scheme, |
| the network location and (part of) the path, to provide missing |
| components in the relative URL. For example: |
| |
| \begin{verbatim} |
| >>> from urlparse import urljoin |
| >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') |
| 'http://www.cwi.nl/%7Eguido/FAQ.html' |
| \end{verbatim} |
| |
| The \var{allow_fragments} argument has the same meaning and default as |
| for \function{urlparse()}. |
| |
| \note{If \var{url} is an absolute URL (that is, starting with \code{//} |
| or \code{scheme://}, the \var{url}'s host name and/or scheme |
| will be present in the result. For example:} |
| |
| \begin{verbatim} |
| >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', |
| ... '//www.python.org/%7Eguido') |
| 'http://www.python.org/%7Eguido' |
| \end{verbatim} |
| |
| If you do not want that behavior, preprocess |
| the \var{url} with \function{urlsplit()} and \function{urlunsplit()}, |
| removing possible \em{scheme} and \em{netloc} parts. |
| \end{funcdesc} |
| |
| \begin{funcdesc}{urldefrag}{url} |
| If \var{url} contains a fragment identifier, returns a modified |
| version of \var{url} with no fragment identifier, and the fragment |
| identifier as a separate string. If there is no fragment identifier |
| in \var{url}, returns \var{url} unmodified and an empty string. |
| \end{funcdesc} |
| |
| |
| \begin{seealso} |
| \seerfc{1738}{Uniform Resource Locators (URL)}{ |
| This specifies the formal syntax and semantics of absolute |
| URLs.} |
| \seerfc{1808}{Relative Uniform Resource Locators}{ |
| This Request For Comments includes the rules for joining an |
| absolute and a relative URL, including a fair number of |
| ``Abnormal Examples'' which govern the treatment of border |
| cases.} |
| \seerfc{2396}{Uniform Resource Identifiers (URI): Generic Syntax}{ |
| Document describing the generic syntactic requirements for |
| both Uniform Resource Names (URNs) and Uniform Resource |
| Locators (URLs).} |
| \end{seealso} |
| |
| |
| \subsection{Results of \function{urlparse()} and \function{urlsplit()} |
| \label{urlparse-result-object}} |
| |
| The result objects from the \function{urlparse()} and |
| \function{urlsplit()} functions are subclasses of the \pytype{tuple} |
| type. These subclasses add the attributes described in those |
| functions, as well as provide an additional method: |
| |
| \begin{methoddesc}[ParseResult]{geturl}{} |
| Return the re-combined version of the original URL as a string. |
| This may differ from the original URL in that the scheme will always |
| be normalized to lower case and empty components may be dropped. |
| Specifically, empty parameters, queries, and fragment identifiers |
| will be removed. |
| |
| The result of this method is a fixpoint if passed back through the |
| original parsing function: |
| |
| \begin{verbatim} |
| >>> import urlparse |
| >>> url = 'HTTP://www.Python.org/doc/#' |
| |
| >>> r1 = urlparse.urlsplit(url) |
| >>> r1.geturl() |
| 'http://www.Python.org/doc/' |
| |
| >>> r2 = urlparse.urlsplit(r1.geturl()) |
| >>> r2.geturl() |
| 'http://www.Python.org/doc/' |
| \end{verbatim} |
| |
| \versionadded{2.5} |
| \end{methoddesc} |
| |
| The following classes provide the implementations of the parse results:: |
| |
| \begin{classdesc*}{BaseResult} |
| Base class for the concrete result classes. This provides most of |
| the attribute definitions. It does not provide a \method{geturl()} |
| method. It is derived from \class{tuple}, but does not override the |
| \method{__init__()} or \method{__new__()} methods. |
| \end{classdesc*} |
| |
| |
| \begin{classdesc}{ParseResult}{scheme, netloc, path, params, query, fragment} |
| Concrete class for \function{urlparse()} results. The |
| \method{__new__()} method is overridden to support checking that the |
| right number of arguments are passed. |
| \end{classdesc} |
| |
| |
| \begin{classdesc}{SplitResult}{scheme, netloc, path, query, fragment} |
| Concrete class for \function{urlsplit()} results. The |
| \method{__new__()} method is overridden to support checking that the |
| right number of arguments are passed. |
| \end{classdesc} |