Christian Heimes | 2202f87 | 2008-02-06 14:31:34 +0000 | [diff] [blame] | 1 | .. _regex-howto: |
| 2 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 3 | **************************** |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 4 | Regular Expression HOWTO |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 5 | **************************** |
| 6 | |
Benjamin Peterson | f07d002 | 2009-03-21 17:31:58 +0000 | [diff] [blame] | 7 | :Author: A.M. Kuchling <amk@amk.ca> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 8 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 9 | .. TODO: |
| 10 | Document lookbehind assertions |
| 11 | Better way of displaying a RE, a string, and what it matches |
| 12 | Mention optional argument to match.groups() |
| 13 | Unicode (at least a reference) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 14 | |
| 15 | |
| 16 | .. topic:: Abstract |
| 17 | |
| 18 | This document is an introductory tutorial to using regular expressions in Python |
| 19 | with the :mod:`re` module. It provides a gentler introduction than the |
| 20 | corresponding section in the Library Reference. |
| 21 | |
| 22 | |
| 23 | Introduction |
| 24 | ============ |
| 25 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 26 | Regular expressions (called REs, or regexes, or regex patterns) are essentially |
| 27 | a tiny, highly specialized programming language embedded inside Python and made |
| 28 | available through the :mod:`re` module. Using this little language, you specify |
| 29 | the rules for the set of possible strings that you want to match; this set might |
| 30 | contain English sentences, or e-mail addresses, or TeX commands, or anything you |
| 31 | like. You can then ask questions such as "Does this string match the pattern?", |
| 32 | or "Is there a match for the pattern anywhere in this string?". You can also |
| 33 | use REs to modify a string or to split it apart in various ways. |
| 34 | |
| 35 | Regular expression patterns are compiled into a series of bytecodes which are |
| 36 | then executed by a matching engine written in C. For advanced use, it may be |
| 37 | necessary to pay careful attention to how the engine will execute a given RE, |
| 38 | and write the RE in a certain way in order to produce bytecode that runs faster. |
| 39 | Optimization isn't covered in this document, because it requires that you have a |
| 40 | good understanding of the matching engine's internals. |
| 41 | |
| 42 | The regular expression language is relatively small and restricted, so not all |
| 43 | possible string processing tasks can be done using regular expressions. There |
| 44 | are also tasks that *can* be done with regular expressions, but the expressions |
| 45 | turn out to be very complicated. In these cases, you may be better off writing |
| 46 | Python code to do the processing; while Python code will be slower than an |
| 47 | elaborate regular expression, it will also probably be more understandable. |
| 48 | |
| 49 | |
| 50 | Simple Patterns |
| 51 | =============== |
| 52 | |
| 53 | We'll start by learning about the simplest possible regular expressions. Since |
| 54 | regular expressions are used to operate on strings, we'll begin with the most |
| 55 | common task: matching characters. |
| 56 | |
| 57 | For a detailed explanation of the computer science underlying regular |
| 58 | expressions (deterministic and non-deterministic finite automata), you can refer |
| 59 | to almost any textbook on writing compilers. |
| 60 | |
| 61 | |
| 62 | Matching Characters |
| 63 | ------------------- |
| 64 | |
| 65 | Most letters and characters will simply match themselves. For example, the |
| 66 | regular expression ``test`` will match the string ``test`` exactly. (You can |
| 67 | enable a case-insensitive mode that would let this RE match ``Test`` or ``TEST`` |
| 68 | as well; more about this later.) |
| 69 | |
| 70 | There are exceptions to this rule; some characters are special |
| 71 | :dfn:`metacharacters`, and don't match themselves. Instead, they signal that |
| 72 | some out-of-the-ordinary thing should be matched, or they affect other portions |
| 73 | of the RE by repeating them or changing their meaning. Much of this document is |
| 74 | devoted to discussing various metacharacters and what they do. |
| 75 | |
| 76 | Here's a complete list of the metacharacters; their meanings will be discussed |
| 77 | in the rest of this HOWTO. :: |
| 78 | |
Georg Brandl | a2388be | 2011-03-06 11:07:11 +0100 | [diff] [blame] | 79 | . ^ $ * + ? { } [ ] \ | ( ) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 80 | |
| 81 | The first metacharacters we'll look at are ``[`` and ``]``. They're used for |
| 82 | specifying a character class, which is a set of characters that you wish to |
| 83 | match. Characters can be listed individually, or a range of characters can be |
| 84 | indicated by giving two characters and separating them by a ``'-'``. For |
| 85 | example, ``[abc]`` will match any of the characters ``a``, ``b``, or ``c``; this |
| 86 | is the same as ``[a-c]``, which uses a range to express the same set of |
| 87 | characters. If you wanted to match only lowercase letters, your RE would be |
| 88 | ``[a-z]``. |
| 89 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 90 | Metacharacters are not active inside classes. For example, ``[akm$]`` will |
| 91 | match any of the characters ``'a'``, ``'k'``, ``'m'``, or ``'$'``; ``'$'`` is |
| 92 | usually a metacharacter, but inside a character class it's stripped of its |
| 93 | special nature. |
| 94 | |
| 95 | You can match the characters not listed within the class by :dfn:`complementing` |
| 96 | the set. This is indicated by including a ``'^'`` as the first character of the |
| 97 | class; ``'^'`` outside a character class will simply match the ``'^'`` |
| 98 | character. For example, ``[^5]`` will match any character except ``'5'``. |
| 99 | |
| 100 | Perhaps the most important metacharacter is the backslash, ``\``. As in Python |
| 101 | string literals, the backslash can be followed by various characters to signal |
| 102 | various special sequences. It's also used to escape all the metacharacters so |
| 103 | you can still match them in patterns; for example, if you need to match a ``[`` |
| 104 | or ``\``, you can precede them with a backslash to remove their special |
| 105 | meaning: ``\[`` or ``\\``. |
| 106 | |
| 107 | Some of the special sequences beginning with ``'\'`` represent predefined sets |
| 108 | of characters that are often useful, such as the set of digits, the set of |
| 109 | letters, or the set of anything that isn't whitespace. The following predefined |
Terry Reedy | 23f4fb9 | 2011-01-10 23:16:24 +0000 | [diff] [blame] | 110 | special sequences are a subset of those available. The equivalent classes are |
| 111 | for bytes patterns. For a complete list of sequences and expanded class |
| 112 | definitions for Unicode string patterns, see the last part of |
| 113 | :ref:`Regular Expression Syntax <re-syntax>`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 114 | |
| 115 | ``\d`` |
| 116 | Matches any decimal digit; this is equivalent to the class ``[0-9]``. |
| 117 | |
| 118 | ``\D`` |
| 119 | Matches any non-digit character; this is equivalent to the class ``[^0-9]``. |
| 120 | |
| 121 | ``\s`` |
| 122 | Matches any whitespace character; this is equivalent to the class ``[ |
| 123 | \t\n\r\f\v]``. |
| 124 | |
| 125 | ``\S`` |
| 126 | Matches any non-whitespace character; this is equivalent to the class ``[^ |
| 127 | \t\n\r\f\v]``. |
| 128 | |
| 129 | ``\w`` |
| 130 | Matches any alphanumeric character; this is equivalent to the class |
| 131 | ``[a-zA-Z0-9_]``. |
| 132 | |
| 133 | ``\W`` |
| 134 | Matches any non-alphanumeric character; this is equivalent to the class |
| 135 | ``[^a-zA-Z0-9_]``. |
| 136 | |
| 137 | These sequences can be included inside a character class. For example, |
| 138 | ``[\s,.]`` is a character class that will match any whitespace character, or |
| 139 | ``','`` or ``'.'``. |
| 140 | |
| 141 | The final metacharacter in this section is ``.``. It matches anything except a |
| 142 | newline character, and there's an alternate mode (``re.DOTALL``) where it will |
| 143 | match even a newline. ``'.'`` is often used where you want to match "any |
| 144 | character". |
| 145 | |
| 146 | |
| 147 | Repeating Things |
| 148 | ---------------- |
| 149 | |
| 150 | Being able to match varying sets of characters is the first thing regular |
| 151 | expressions can do that isn't already possible with the methods available on |
| 152 | strings. However, if that was the only additional capability of regexes, they |
| 153 | wouldn't be much of an advance. Another capability is that you can specify that |
| 154 | portions of the RE must be repeated a certain number of times. |
| 155 | |
| 156 | The first metacharacter for repeating things that we'll look at is ``*``. ``*`` |
| 157 | doesn't match the literal character ``*``; instead, it specifies that the |
| 158 | previous character can be matched zero or more times, instead of exactly once. |
| 159 | |
| 160 | For example, ``ca*t`` will match ``ct`` (0 ``a`` characters), ``cat`` (1 ``a``), |
| 161 | ``caaat`` (3 ``a`` characters), and so forth. The RE engine has various |
| 162 | internal limitations stemming from the size of C's ``int`` type that will |
| 163 | prevent it from matching over 2 billion ``a`` characters; you probably don't |
| 164 | have enough memory to construct a string that large, so you shouldn't run into |
| 165 | that limit. |
| 166 | |
| 167 | Repetitions such as ``*`` are :dfn:`greedy`; when repeating a RE, the matching |
| 168 | engine will try to repeat it as many times as possible. If later portions of the |
| 169 | pattern don't match, the matching engine will then back up and try again with |
| 170 | few repetitions. |
| 171 | |
| 172 | A step-by-step example will make this more obvious. Let's consider the |
| 173 | expression ``a[bcd]*b``. This matches the letter ``'a'``, zero or more letters |
| 174 | from the class ``[bcd]``, and finally ends with a ``'b'``. Now imagine matching |
| 175 | this RE against the string ``abcbd``. |
| 176 | |
| 177 | +------+-----------+---------------------------------+ |
| 178 | | Step | Matched | Explanation | |
| 179 | +======+===========+=================================+ |
| 180 | | 1 | ``a`` | The ``a`` in the RE matches. | |
| 181 | +------+-----------+---------------------------------+ |
| 182 | | 2 | ``abcbd`` | The engine matches ``[bcd]*``, | |
| 183 | | | | going as far as it can, which | |
| 184 | | | | is to the end of the string. | |
| 185 | +------+-----------+---------------------------------+ |
| 186 | | 3 | *Failure* | The engine tries to match | |
| 187 | | | | ``b``, but the current position | |
| 188 | | | | is at the end of the string, so | |
| 189 | | | | it fails. | |
| 190 | +------+-----------+---------------------------------+ |
| 191 | | 4 | ``abcb`` | Back up, so that ``[bcd]*`` | |
| 192 | | | | matches one less character. | |
| 193 | +------+-----------+---------------------------------+ |
| 194 | | 5 | *Failure* | Try ``b`` again, but the | |
| 195 | | | | current position is at the last | |
| 196 | | | | character, which is a ``'d'``. | |
| 197 | +------+-----------+---------------------------------+ |
| 198 | | 6 | ``abc`` | Back up again, so that | |
| 199 | | | | ``[bcd]*`` is only matching | |
| 200 | | | | ``bc``. | |
| 201 | +------+-----------+---------------------------------+ |
| 202 | | 6 | ``abcb`` | Try ``b`` again. This time | |
Christian Heimes | a612dc0 | 2008-02-24 13:08:18 +0000 | [diff] [blame] | 203 | | | | the character at the | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 204 | | | | current position is ``'b'``, so | |
| 205 | | | | it succeeds. | |
| 206 | +------+-----------+---------------------------------+ |
| 207 | |
| 208 | The end of the RE has now been reached, and it has matched ``abcb``. This |
| 209 | demonstrates how the matching engine goes as far as it can at first, and if no |
| 210 | match is found it will then progressively back up and retry the rest of the RE |
| 211 | again and again. It will back up until it has tried zero matches for |
| 212 | ``[bcd]*``, and if that subsequently fails, the engine will conclude that the |
| 213 | string doesn't match the RE at all. |
| 214 | |
| 215 | Another repeating metacharacter is ``+``, which matches one or more times. Pay |
| 216 | careful attention to the difference between ``*`` and ``+``; ``*`` matches |
| 217 | *zero* or more times, so whatever's being repeated may not be present at all, |
| 218 | while ``+`` requires at least *one* occurrence. To use a similar example, |
| 219 | ``ca+t`` will match ``cat`` (1 ``a``), ``caaat`` (3 ``a``'s), but won't match |
| 220 | ``ct``. |
| 221 | |
| 222 | There are two more repeating qualifiers. The question mark character, ``?``, |
| 223 | matches either once or zero times; you can think of it as marking something as |
| 224 | being optional. For example, ``home-?brew`` matches either ``homebrew`` or |
| 225 | ``home-brew``. |
| 226 | |
| 227 | The most complicated repeated qualifier is ``{m,n}``, where *m* and *n* are |
| 228 | decimal integers. This qualifier means there must be at least *m* repetitions, |
| 229 | and at most *n*. For example, ``a/{1,3}b`` will match ``a/b``, ``a//b``, and |
| 230 | ``a///b``. It won't match ``ab``, which has no slashes, or ``a////b``, which |
| 231 | has four. |
| 232 | |
| 233 | You can omit either *m* or *n*; in that case, a reasonable value is assumed for |
| 234 | the missing value. Omitting *m* is interpreted as a lower limit of 0, while |
| 235 | omitting *n* results in an upper bound of infinity --- actually, the upper bound |
| 236 | is the 2-billion limit mentioned earlier, but that might as well be infinity. |
| 237 | |
| 238 | Readers of a reductionist bent may notice that the three other qualifiers can |
| 239 | all be expressed using this notation. ``{0,}`` is the same as ``*``, ``{1,}`` |
| 240 | is equivalent to ``+``, and ``{0,1}`` is the same as ``?``. It's better to use |
| 241 | ``*``, ``+``, or ``?`` when you can, simply because they're shorter and easier |
| 242 | to read. |
| 243 | |
| 244 | |
| 245 | Using Regular Expressions |
| 246 | ========================= |
| 247 | |
| 248 | Now that we've looked at some simple regular expressions, how do we actually use |
| 249 | them in Python? The :mod:`re` module provides an interface to the regular |
| 250 | expression engine, allowing you to compile REs into objects and then perform |
| 251 | matches with them. |
| 252 | |
| 253 | |
| 254 | Compiling Regular Expressions |
| 255 | ----------------------------- |
| 256 | |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 257 | Regular expressions are compiled into pattern objects, which have |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 258 | methods for various operations such as searching for pattern matches or |
| 259 | performing string substitutions. :: |
| 260 | |
| 261 | >>> import re |
| 262 | >>> p = re.compile('ab*') |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 263 | >>> p |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 264 | <_sre.SRE_Pattern object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 265 | |
| 266 | :func:`re.compile` also accepts an optional *flags* argument, used to enable |
| 267 | various special features and syntax variations. We'll go over the available |
| 268 | settings later, but for now a single example will do:: |
| 269 | |
| 270 | >>> p = re.compile('ab*', re.IGNORECASE) |
| 271 | |
| 272 | The RE is passed to :func:`re.compile` as a string. REs are handled as strings |
| 273 | because regular expressions aren't part of the core Python language, and no |
| 274 | special syntax was created for expressing them. (There are applications that |
| 275 | don't need REs at all, so there's no need to bloat the language specification by |
| 276 | including them.) Instead, the :mod:`re` module is simply a C extension module |
| 277 | included with Python, just like the :mod:`socket` or :mod:`zlib` modules. |
| 278 | |
| 279 | Putting REs in strings keeps the Python language simpler, but has one |
| 280 | disadvantage which is the topic of the next section. |
| 281 | |
| 282 | |
| 283 | The Backslash Plague |
| 284 | -------------------- |
| 285 | |
| 286 | As stated earlier, regular expressions use the backslash character (``'\'``) to |
| 287 | indicate special forms or to allow special characters to be used without |
| 288 | invoking their special meaning. This conflicts with Python's usage of the same |
| 289 | character for the same purpose in string literals. |
| 290 | |
| 291 | Let's say you want to write a RE that matches the string ``\section``, which |
| 292 | might be found in a LaTeX file. To figure out what to write in the program |
| 293 | code, start with the desired string to be matched. Next, you must escape any |
| 294 | backslashes and other metacharacters by preceding them with a backslash, |
| 295 | resulting in the string ``\\section``. The resulting string that must be passed |
| 296 | to :func:`re.compile` must be ``\\section``. However, to express this as a |
| 297 | Python string literal, both backslashes must be escaped *again*. |
| 298 | |
| 299 | +-------------------+------------------------------------------+ |
| 300 | | Characters | Stage | |
| 301 | +===================+==========================================+ |
| 302 | | ``\section`` | Text string to be matched | |
| 303 | +-------------------+------------------------------------------+ |
| 304 | | ``\\section`` | Escaped backslash for :func:`re.compile` | |
| 305 | +-------------------+------------------------------------------+ |
| 306 | | ``"\\\\section"`` | Escaped backslashes for a string literal | |
| 307 | +-------------------+------------------------------------------+ |
| 308 | |
| 309 | In short, to match a literal backslash, one has to write ``'\\\\'`` as the RE |
| 310 | string, because the regular expression must be ``\\``, and each backslash must |
| 311 | be expressed as ``\\`` inside a regular Python string literal. In REs that |
| 312 | feature backslashes repeatedly, this leads to lots of repeated backslashes and |
| 313 | makes the resulting strings difficult to understand. |
| 314 | |
| 315 | The solution is to use Python's raw string notation for regular expressions; |
| 316 | backslashes are not handled in any special way in a string literal prefixed with |
| 317 | ``'r'``, so ``r"\n"`` is a two-character string containing ``'\'`` and ``'n'``, |
| 318 | while ``"\n"`` is a one-character string containing a newline. Regular |
| 319 | expressions will often be written in Python code using this raw string notation. |
| 320 | |
| 321 | +-------------------+------------------+ |
| 322 | | Regular String | Raw string | |
| 323 | +===================+==================+ |
| 324 | | ``"ab*"`` | ``r"ab*"`` | |
| 325 | +-------------------+------------------+ |
| 326 | | ``"\\\\section"`` | ``r"\\section"`` | |
| 327 | +-------------------+------------------+ |
| 328 | | ``"\\w+\\s+\\1"`` | ``r"\w+\s+\1"`` | |
| 329 | +-------------------+------------------+ |
| 330 | |
| 331 | |
| 332 | Performing Matches |
| 333 | ------------------ |
| 334 | |
| 335 | Once you have an object representing a compiled regular expression, what do you |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 336 | do with it? Pattern objects have several methods and attributes. |
Georg Brandl | 86def6c | 2008-01-21 20:36:10 +0000 | [diff] [blame] | 337 | Only the most significant ones will be covered here; consult the :mod:`re` docs |
| 338 | for a complete listing. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 339 | |
| 340 | +------------------+-----------------------------------------------+ |
| 341 | | Method/Attribute | Purpose | |
| 342 | +==================+===============================================+ |
| 343 | | ``match()`` | Determine if the RE matches at the beginning | |
| 344 | | | of the string. | |
| 345 | +------------------+-----------------------------------------------+ |
| 346 | | ``search()`` | Scan through a string, looking for any | |
| 347 | | | location where this RE matches. | |
| 348 | +------------------+-----------------------------------------------+ |
| 349 | | ``findall()`` | Find all substrings where the RE matches, and | |
| 350 | | | returns them as a list. | |
| 351 | +------------------+-----------------------------------------------+ |
| 352 | | ``finditer()`` | Find all substrings where the RE matches, and | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 353 | | | returns them as an :term:`iterator`. | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 354 | +------------------+-----------------------------------------------+ |
| 355 | |
| 356 | :meth:`match` and :meth:`search` return ``None`` if no match can be found. If |
| 357 | they're successful, a ``MatchObject`` instance is returned, containing |
| 358 | information about the match: where it starts and ends, the substring it matched, |
| 359 | and more. |
| 360 | |
| 361 | You can learn about this by interactively experimenting with the :mod:`re` |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 362 | module. If you have :mod:`tkinter` available, you may also want to look at |
Éric Araujo | fdfaf0a | 2012-03-05 15:50:37 +0100 | [diff] [blame] | 363 | :source:`Tools/demo/redemo.py`, a demonstration program included with the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 364 | Python distribution. It allows you to enter REs and strings, and displays |
| 365 | whether the RE matches or fails. :file:`redemo.py` can be quite useful when |
| 366 | trying to debug a complicated RE. Phil Schwartz's `Kodos |
Christian Heimes | dd15f6c | 2008-03-16 00:07:10 +0000 | [diff] [blame] | 367 | <http://kodos.sourceforge.net/>`_ is also an interactive tool for developing and |
| 368 | testing RE patterns. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 369 | |
| 370 | This HOWTO uses the standard Python interpreter for its examples. First, run the |
| 371 | Python interpreter, import the :mod:`re` module, and compile a RE:: |
| 372 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 373 | >>> import re |
| 374 | >>> p = re.compile('[a-z]+') |
| 375 | >>> p |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 376 | <_sre.SRE_Pattern object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 377 | |
| 378 | Now, you can try matching various strings against the RE ``[a-z]+``. An empty |
| 379 | string shouldn't match at all, since ``+`` means 'one or more repetitions'. |
| 380 | :meth:`match` should return ``None`` in this case, which will cause the |
| 381 | interpreter to print no output. You can explicitly print the result of |
| 382 | :meth:`match` to make this clear. :: |
| 383 | |
| 384 | >>> p.match("") |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 385 | >>> print(p.match("")) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 386 | None |
| 387 | |
| 388 | Now, let's try it on a string that it should match, such as ``tempo``. In this |
| 389 | case, :meth:`match` will return a :class:`MatchObject`, so you should store the |
| 390 | result in a variable for later use. :: |
| 391 | |
| 392 | >>> m = p.match('tempo') |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 393 | >>> m |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 394 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 395 | |
| 396 | Now you can query the :class:`MatchObject` for information about the matching |
| 397 | string. :class:`MatchObject` instances also have several methods and |
| 398 | attributes; the most important ones are: |
| 399 | |
| 400 | +------------------+--------------------------------------------+ |
| 401 | | Method/Attribute | Purpose | |
| 402 | +==================+============================================+ |
| 403 | | ``group()`` | Return the string matched by the RE | |
| 404 | +------------------+--------------------------------------------+ |
| 405 | | ``start()`` | Return the starting position of the match | |
| 406 | +------------------+--------------------------------------------+ |
| 407 | | ``end()`` | Return the ending position of the match | |
| 408 | +------------------+--------------------------------------------+ |
| 409 | | ``span()`` | Return a tuple containing the (start, end) | |
| 410 | | | positions of the match | |
| 411 | +------------------+--------------------------------------------+ |
| 412 | |
| 413 | Trying these methods will soon clarify their meaning:: |
| 414 | |
| 415 | >>> m.group() |
| 416 | 'tempo' |
| 417 | >>> m.start(), m.end() |
| 418 | (0, 5) |
| 419 | >>> m.span() |
| 420 | (0, 5) |
| 421 | |
| 422 | :meth:`group` returns the substring that was matched by the RE. :meth:`start` |
| 423 | and :meth:`end` return the starting and ending index of the match. :meth:`span` |
| 424 | returns both start and end indexes in a single tuple. Since the :meth:`match` |
| 425 | method only checks if the RE matches at the start of a string, :meth:`start` |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 426 | will always be zero. However, the :meth:`search` method of patterns |
| 427 | scans through the string, so the match may not start at zero in that |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 428 | case. :: |
| 429 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 430 | >>> print(p.match('::: message')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 431 | None |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 432 | >>> m = p.search('::: message') ; print(m) |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 433 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 434 | >>> m.group() |
| 435 | 'message' |
| 436 | >>> m.span() |
| 437 | (4, 11) |
| 438 | |
| 439 | In actual programs, the most common style is to store the :class:`MatchObject` |
| 440 | in a variable, and then check if it was ``None``. This usually looks like:: |
| 441 | |
| 442 | p = re.compile( ... ) |
| 443 | m = p.match( 'string goes here' ) |
| 444 | if m: |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 445 | print('Match found: ', m.group()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 446 | else: |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 447 | print('No match') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 448 | |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 449 | Two pattern methods return all of the matches for a pattern. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 450 | :meth:`findall` returns a list of matching strings:: |
| 451 | |
| 452 | >>> p = re.compile('\d+') |
| 453 | >>> p.findall('12 drummers drumming, 11 pipers piping, 10 lords a-leaping') |
| 454 | ['12', '11', '10'] |
| 455 | |
| 456 | :meth:`findall` has to create the entire list before it can be returned as the |
| 457 | result. The :meth:`finditer` method returns a sequence of :class:`MatchObject` |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 458 | instances as an :term:`iterator`:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 459 | |
| 460 | >>> iterator = p.finditer('12 drummers drumming, 11 ... 10 ...') |
| 461 | >>> iterator |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 462 | <callable_iterator object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 463 | >>> for match in iterator: |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 464 | ... print(match.span()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 465 | ... |
| 466 | (0, 2) |
| 467 | (22, 24) |
| 468 | (29, 31) |
| 469 | |
| 470 | |
| 471 | Module-Level Functions |
| 472 | ---------------------- |
| 473 | |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 474 | You don't have to create a pattern object and call its methods; the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 475 | :mod:`re` module also provides top-level functions called :func:`match`, |
| 476 | :func:`search`, :func:`findall`, :func:`sub`, and so forth. These functions |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 477 | take the same arguments as the corresponding pattern method, with |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 478 | the RE string added as the first argument, and still return either ``None`` or a |
| 479 | :class:`MatchObject` instance. :: |
| 480 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 481 | >>> print(re.match(r'From\s+', 'Fromage amk')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 482 | None |
| 483 | >>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998') |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 484 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 485 | |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 486 | Under the hood, these functions simply create a pattern object for you |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 487 | and call the appropriate method on it. They also store the compiled object in a |
| 488 | cache, so future calls using the same RE are faster. |
| 489 | |
| 490 | Should you use these module-level functions, or should you get the |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 491 | pattern and call its methods yourself? That choice depends on how |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 492 | frequently the RE will be used, and on your personal coding style. If the RE is |
| 493 | being used at only one point in the code, then the module functions are probably |
| 494 | more convenient. If a program contains a lot of regular expressions, or re-uses |
| 495 | the same ones in several locations, then it might be worthwhile to collect all |
| 496 | the definitions in one place, in a section of code that compiles all the REs |
| 497 | ahead of time. To take an example from the standard library, here's an extract |
Éric Araujo | fdfaf0a | 2012-03-05 15:50:37 +0100 | [diff] [blame] | 498 | from the now-defunct Python 2 standard :mod:`xmllib` module:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 499 | |
| 500 | ref = re.compile( ... ) |
| 501 | entityref = re.compile( ... ) |
| 502 | charref = re.compile( ... ) |
| 503 | starttagopen = re.compile( ... ) |
| 504 | |
| 505 | I generally prefer to work with the compiled object, even for one-time uses, but |
| 506 | few people will be as much of a purist about this as I am. |
| 507 | |
| 508 | |
| 509 | Compilation Flags |
| 510 | ----------------- |
| 511 | |
| 512 | Compilation flags let you modify some aspects of how regular expressions work. |
| 513 | Flags are available in the :mod:`re` module under two names, a long name such as |
| 514 | :const:`IGNORECASE` and a short, one-letter form such as :const:`I`. (If you're |
| 515 | familiar with Perl's pattern modifiers, the one-letter forms use the same |
| 516 | letters; the short form of :const:`re.VERBOSE` is :const:`re.X`, for example.) |
| 517 | Multiple flags can be specified by bitwise OR-ing them; ``re.I | re.M`` sets |
| 518 | both the :const:`I` and :const:`M` flags, for example. |
| 519 | |
| 520 | Here's a table of the available flags, followed by a more detailed explanation |
| 521 | of each one. |
| 522 | |
| 523 | +---------------------------------+--------------------------------------------+ |
| 524 | | Flag | Meaning | |
| 525 | +=================================+============================================+ |
| 526 | | :const:`DOTALL`, :const:`S` | Make ``.`` match any character, including | |
| 527 | | | newlines | |
| 528 | +---------------------------------+--------------------------------------------+ |
| 529 | | :const:`IGNORECASE`, :const:`I` | Do case-insensitive matches | |
| 530 | +---------------------------------+--------------------------------------------+ |
| 531 | | :const:`LOCALE`, :const:`L` | Do a locale-aware match | |
| 532 | +---------------------------------+--------------------------------------------+ |
| 533 | | :const:`MULTILINE`, :const:`M` | Multi-line matching, affecting ``^`` and | |
| 534 | | | ``$`` | |
| 535 | +---------------------------------+--------------------------------------------+ |
| 536 | | :const:`VERBOSE`, :const:`X` | Enable verbose REs, which can be organized | |
| 537 | | | more cleanly and understandably. | |
| 538 | +---------------------------------+--------------------------------------------+ |
Georg Brandl | ce9fbd3 | 2009-03-31 18:41:03 +0000 | [diff] [blame] | 539 | | :const:`ASCII`, :const:`A` | Makes several escapes like ``\w``, ``\b``, | |
| 540 | | | ``\s`` and ``\d`` match only on ASCII | |
| 541 | | | characters with the respective property. | |
| 542 | +---------------------------------+--------------------------------------------+ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 543 | |
| 544 | |
| 545 | .. data:: I |
| 546 | IGNORECASE |
| 547 | :noindex: |
| 548 | |
| 549 | Perform case-insensitive matching; character class and literal strings will |
| 550 | match letters by ignoring case. For example, ``[A-Z]`` will match lowercase |
| 551 | letters, too, and ``Spam`` will match ``Spam``, ``spam``, or ``spAM``. This |
| 552 | lowercasing doesn't take the current locale into account; it will if you also |
| 553 | set the :const:`LOCALE` flag. |
| 554 | |
| 555 | |
| 556 | .. data:: L |
| 557 | LOCALE |
| 558 | :noindex: |
| 559 | |
| 560 | Make ``\w``, ``\W``, ``\b``, and ``\B``, dependent on the current locale. |
| 561 | |
| 562 | Locales are a feature of the C library intended to help in writing programs that |
| 563 | take account of language differences. For example, if you're processing French |
| 564 | text, you'd want to be able to write ``\w+`` to match words, but ``\w`` only |
| 565 | matches the character class ``[A-Za-z]``; it won't match ``'é'`` or ``'ç'``. If |
| 566 | your system is configured properly and a French locale is selected, certain C |
| 567 | functions will tell the program that ``'é'`` should also be considered a letter. |
| 568 | Setting the :const:`LOCALE` flag when compiling a regular expression will cause |
| 569 | the resulting compiled object to use these C functions for ``\w``; this is |
| 570 | slower, but also enables ``\w+`` to match French words as you'd expect. |
| 571 | |
| 572 | |
| 573 | .. data:: M |
| 574 | MULTILINE |
| 575 | :noindex: |
| 576 | |
| 577 | (``^`` and ``$`` haven't been explained yet; they'll be introduced in section |
| 578 | :ref:`more-metacharacters`.) |
| 579 | |
| 580 | Usually ``^`` matches only at the beginning of the string, and ``$`` matches |
| 581 | only at the end of the string and immediately before the newline (if any) at the |
| 582 | end of the string. When this flag is specified, ``^`` matches at the beginning |
| 583 | of the string and at the beginning of each line within the string, immediately |
| 584 | following each newline. Similarly, the ``$`` metacharacter matches either at |
| 585 | the end of the string and at the end of each line (immediately preceding each |
| 586 | newline). |
| 587 | |
| 588 | |
| 589 | .. data:: S |
| 590 | DOTALL |
| 591 | :noindex: |
| 592 | |
| 593 | Makes the ``'.'`` special character match any character at all, including a |
| 594 | newline; without this flag, ``'.'`` will match anything *except* a newline. |
| 595 | |
| 596 | |
Georg Brandl | ce9fbd3 | 2009-03-31 18:41:03 +0000 | [diff] [blame] | 597 | .. data:: A |
| 598 | ASCII |
| 599 | :noindex: |
| 600 | |
| 601 | Make ``\w``, ``\W``, ``\b``, ``\B``, ``\s`` and ``\S`` perform ASCII-only |
| 602 | matching instead of full Unicode matching. This is only meaningful for |
| 603 | Unicode patterns, and is ignored for byte patterns. |
| 604 | |
| 605 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 606 | .. data:: X |
| 607 | VERBOSE |
| 608 | :noindex: |
| 609 | |
| 610 | This flag allows you to write regular expressions that are more readable by |
| 611 | granting you more flexibility in how you can format them. When this flag has |
| 612 | been specified, whitespace within the RE string is ignored, except when the |
| 613 | whitespace is in a character class or preceded by an unescaped backslash; this |
| 614 | lets you organize and indent the RE more clearly. This flag also lets you put |
| 615 | comments within a RE that will be ignored by the engine; comments are marked by |
| 616 | a ``'#'`` that's neither in a character class or preceded by an unescaped |
| 617 | backslash. |
| 618 | |
| 619 | For example, here's a RE that uses :const:`re.VERBOSE`; see how much easier it |
| 620 | is to read? :: |
| 621 | |
| 622 | charref = re.compile(r""" |
Georg Brandl | 06788c9 | 2009-01-03 21:31:47 +0000 | [diff] [blame] | 623 | &[#] # Start of a numeric entity reference |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 624 | ( |
| 625 | 0[0-7]+ # Octal form |
| 626 | | [0-9]+ # Decimal form |
| 627 | | x[0-9a-fA-F]+ # Hexadecimal form |
| 628 | ) |
| 629 | ; # Trailing semicolon |
| 630 | """, re.VERBOSE) |
| 631 | |
| 632 | Without the verbose setting, the RE would look like this:: |
| 633 | |
| 634 | charref = re.compile("&#(0[0-7]+" |
| 635 | "|[0-9]+" |
| 636 | "|x[0-9a-fA-F]+);") |
| 637 | |
| 638 | In the above example, Python's automatic concatenation of string literals has |
| 639 | been used to break up the RE into smaller pieces, but it's still more difficult |
| 640 | to understand than the version using :const:`re.VERBOSE`. |
| 641 | |
| 642 | |
| 643 | More Pattern Power |
| 644 | ================== |
| 645 | |
| 646 | So far we've only covered a part of the features of regular expressions. In |
| 647 | this section, we'll cover some new metacharacters, and how to use groups to |
| 648 | retrieve portions of the text that was matched. |
| 649 | |
| 650 | |
| 651 | .. _more-metacharacters: |
| 652 | |
| 653 | More Metacharacters |
| 654 | ------------------- |
| 655 | |
| 656 | There are some metacharacters that we haven't covered yet. Most of them will be |
| 657 | covered in this section. |
| 658 | |
| 659 | Some of the remaining metacharacters to be discussed are :dfn:`zero-width |
| 660 | assertions`. They don't cause the engine to advance through the string; |
| 661 | instead, they consume no characters at all, and simply succeed or fail. For |
| 662 | example, ``\b`` is an assertion that the current position is located at a word |
| 663 | boundary; the position isn't changed by the ``\b`` at all. This means that |
| 664 | zero-width assertions should never be repeated, because if they match once at a |
| 665 | given location, they can obviously be matched an infinite number of times. |
| 666 | |
| 667 | ``|`` |
| 668 | Alternation, or the "or" operator. If A and B are regular expressions, |
| 669 | ``A|B`` will match any string that matches either ``A`` or ``B``. ``|`` has very |
| 670 | low precedence in order to make it work reasonably when you're alternating |
| 671 | multi-character strings. ``Crow|Servo`` will match either ``Crow`` or ``Servo``, |
| 672 | not ``Cro``, a ``'w'`` or an ``'S'``, and ``ervo``. |
| 673 | |
| 674 | To match a literal ``'|'``, use ``\|``, or enclose it inside a character class, |
| 675 | as in ``[|]``. |
| 676 | |
| 677 | ``^`` |
| 678 | Matches at the beginning of lines. Unless the :const:`MULTILINE` flag has been |
| 679 | set, this will only match at the beginning of the string. In :const:`MULTILINE` |
| 680 | mode, this also matches immediately after each newline within the string. |
| 681 | |
| 682 | For example, if you wish to match the word ``From`` only at the beginning of a |
| 683 | line, the RE to use is ``^From``. :: |
| 684 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 685 | >>> print(re.search('^From', 'From Here to Eternity')) |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 686 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 687 | >>> print(re.search('^From', 'Reciting From Memory')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 688 | None |
| 689 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 690 | .. To match a literal \character{\^}, use \regexp{\e\^} or enclose it |
| 691 | .. inside a character class, as in \regexp{[{\e}\^]}. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 692 | |
| 693 | ``$`` |
| 694 | Matches at the end of a line, which is defined as either the end of the string, |
| 695 | or any location followed by a newline character. :: |
| 696 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 697 | >>> print(re.search('}$', '{block}')) |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 698 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 699 | >>> print(re.search('}$', '{block} ')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 700 | None |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 701 | >>> print(re.search('}$', '{block}\n')) |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 702 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 703 | |
| 704 | To match a literal ``'$'``, use ``\$`` or enclose it inside a character class, |
| 705 | as in ``[$]``. |
| 706 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 707 | ``\A`` |
| 708 | Matches only at the start of the string. When not in :const:`MULTILINE` mode, |
| 709 | ``\A`` and ``^`` are effectively the same. In :const:`MULTILINE` mode, they're |
| 710 | different: ``\A`` still matches only at the beginning of the string, but ``^`` |
| 711 | may match at any location inside the string that follows a newline character. |
| 712 | |
| 713 | ``\Z`` |
| 714 | Matches only at the end of the string. |
| 715 | |
| 716 | ``\b`` |
| 717 | Word boundary. This is a zero-width assertion that matches only at the |
| 718 | beginning or end of a word. A word is defined as a sequence of alphanumeric |
| 719 | characters, so the end of a word is indicated by whitespace or a |
| 720 | non-alphanumeric character. |
| 721 | |
| 722 | The following example matches ``class`` only when it's a complete word; it won't |
| 723 | match when it's contained inside another word. :: |
| 724 | |
| 725 | >>> p = re.compile(r'\bclass\b') |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 726 | >>> print(p.search('no class at all')) |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 727 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 728 | >>> print(p.search('the declassified algorithm')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 729 | None |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 730 | >>> print(p.search('one subclass is')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 731 | None |
| 732 | |
| 733 | There are two subtleties you should remember when using this special sequence. |
| 734 | First, this is the worst collision between Python's string literals and regular |
| 735 | expression sequences. In Python's string literals, ``\b`` is the backspace |
| 736 | character, ASCII value 8. If you're not using raw strings, then Python will |
| 737 | convert the ``\b`` to a backspace, and your RE won't match as you expect it to. |
| 738 | The following example looks the same as our previous RE, but omits the ``'r'`` |
| 739 | in front of the RE string. :: |
| 740 | |
| 741 | >>> p = re.compile('\bclass\b') |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 742 | >>> print(p.search('no class at all')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 743 | None |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 744 | >>> print(p.search('\b' + 'class' + '\b') ) |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 745 | <_sre.SRE_Match object at 0x...> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 746 | |
| 747 | Second, inside a character class, where there's no use for this assertion, |
| 748 | ``\b`` represents the backspace character, for compatibility with Python's |
| 749 | string literals. |
| 750 | |
| 751 | ``\B`` |
| 752 | Another zero-width assertion, this is the opposite of ``\b``, only matching when |
| 753 | the current position is not at a word boundary. |
| 754 | |
| 755 | |
| 756 | Grouping |
| 757 | -------- |
| 758 | |
| 759 | Frequently you need to obtain more information than just whether the RE matched |
| 760 | or not. Regular expressions are often used to dissect strings by writing a RE |
| 761 | divided into several subgroups which match different components of interest. |
| 762 | For example, an RFC-822 header line is divided into a header name and a value, |
| 763 | separated by a ``':'``, like this:: |
| 764 | |
| 765 | From: author@example.com |
| 766 | User-Agent: Thunderbird 1.5.0.9 (X11/20061227) |
| 767 | MIME-Version: 1.0 |
| 768 | To: editor@example.com |
| 769 | |
| 770 | This can be handled by writing a regular expression which matches an entire |
| 771 | header line, and has one group which matches the header name, and another group |
| 772 | which matches the header's value. |
| 773 | |
| 774 | Groups are marked by the ``'('``, ``')'`` metacharacters. ``'('`` and ``')'`` |
| 775 | have much the same meaning as they do in mathematical expressions; they group |
| 776 | together the expressions contained inside them, and you can repeat the contents |
| 777 | of a group with a repeating qualifier, such as ``*``, ``+``, ``?``, or |
| 778 | ``{m,n}``. For example, ``(ab)*`` will match zero or more repetitions of |
| 779 | ``ab``. :: |
| 780 | |
| 781 | >>> p = re.compile('(ab)*') |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 782 | >>> print(p.match('ababababab').span()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 783 | (0, 10) |
| 784 | |
| 785 | Groups indicated with ``'('``, ``')'`` also capture the starting and ending |
| 786 | index of the text that they match; this can be retrieved by passing an argument |
| 787 | to :meth:`group`, :meth:`start`, :meth:`end`, and :meth:`span`. Groups are |
| 788 | numbered starting with 0. Group 0 is always present; it's the whole RE, so |
| 789 | :class:`MatchObject` methods all have group 0 as their default argument. Later |
| 790 | we'll see how to express groups that don't capture the span of text that they |
| 791 | match. :: |
| 792 | |
| 793 | >>> p = re.compile('(a)b') |
| 794 | >>> m = p.match('ab') |
| 795 | >>> m.group() |
| 796 | 'ab' |
| 797 | >>> m.group(0) |
| 798 | 'ab' |
| 799 | |
| 800 | Subgroups are numbered from left to right, from 1 upward. Groups can be nested; |
| 801 | to determine the number, just count the opening parenthesis characters, going |
| 802 | from left to right. :: |
| 803 | |
| 804 | >>> p = re.compile('(a(b)c)d') |
| 805 | >>> m = p.match('abcd') |
| 806 | >>> m.group(0) |
| 807 | 'abcd' |
| 808 | >>> m.group(1) |
| 809 | 'abc' |
| 810 | >>> m.group(2) |
| 811 | 'b' |
| 812 | |
| 813 | :meth:`group` can be passed multiple group numbers at a time, in which case it |
| 814 | will return a tuple containing the corresponding values for those groups. :: |
| 815 | |
| 816 | >>> m.group(2,1,2) |
| 817 | ('b', 'abc', 'b') |
| 818 | |
| 819 | The :meth:`groups` method returns a tuple containing the strings for all the |
| 820 | subgroups, from 1 up to however many there are. :: |
| 821 | |
| 822 | >>> m.groups() |
| 823 | ('abc', 'b') |
| 824 | |
| 825 | Backreferences in a pattern allow you to specify that the contents of an earlier |
| 826 | capturing group must also be found at the current location in the string. For |
| 827 | example, ``\1`` will succeed if the exact contents of group 1 can be found at |
| 828 | the current position, and fails otherwise. Remember that Python's string |
| 829 | literals also use a backslash followed by numbers to allow including arbitrary |
| 830 | characters in a string, so be sure to use a raw string when incorporating |
| 831 | backreferences in a RE. |
| 832 | |
| 833 | For example, the following RE detects doubled words in a string. :: |
| 834 | |
| 835 | >>> p = re.compile(r'(\b\w+)\s+\1') |
| 836 | >>> p.search('Paris in the the spring').group() |
| 837 | 'the the' |
| 838 | |
| 839 | Backreferences like this aren't often useful for just searching through a string |
| 840 | --- there are few text formats which repeat data in this way --- but you'll soon |
| 841 | find out that they're *very* useful when performing string substitutions. |
| 842 | |
| 843 | |
| 844 | Non-capturing and Named Groups |
| 845 | ------------------------------ |
| 846 | |
| 847 | Elaborate REs may use many groups, both to capture substrings of interest, and |
| 848 | to group and structure the RE itself. In complex REs, it becomes difficult to |
| 849 | keep track of the group numbers. There are two features which help with this |
| 850 | problem. Both of them use a common syntax for regular expression extensions, so |
| 851 | we'll look at that first. |
| 852 | |
| 853 | Perl 5 added several additional features to standard regular expressions, and |
| 854 | the Python :mod:`re` module supports most of them. It would have been |
| 855 | difficult to choose new single-keystroke metacharacters or new special sequences |
| 856 | beginning with ``\`` to represent the new features without making Perl's regular |
| 857 | expressions confusingly different from standard REs. If you chose ``&`` as a |
| 858 | new metacharacter, for example, old expressions would be assuming that ``&`` was |
| 859 | a regular character and wouldn't have escaped it by writing ``\&`` or ``[&]``. |
| 860 | |
| 861 | The solution chosen by the Perl developers was to use ``(?...)`` as the |
| 862 | extension syntax. ``?`` immediately after a parenthesis was a syntax error |
| 863 | because the ``?`` would have nothing to repeat, so this didn't introduce any |
| 864 | compatibility problems. The characters immediately after the ``?`` indicate |
| 865 | what extension is being used, so ``(?=foo)`` is one thing (a positive lookahead |
| 866 | assertion) and ``(?:foo)`` is something else (a non-capturing group containing |
| 867 | the subexpression ``foo``). |
| 868 | |
| 869 | Python adds an extension syntax to Perl's extension syntax. If the first |
| 870 | character after the question mark is a ``P``, you know that it's an extension |
| 871 | that's specific to Python. Currently there are two such extensions: |
| 872 | ``(?P<name>...)`` defines a named group, and ``(?P=name)`` is a backreference to |
| 873 | a named group. If future versions of Perl 5 add similar features using a |
| 874 | different syntax, the :mod:`re` module will be changed to support the new |
| 875 | syntax, while preserving the Python-specific syntax for compatibility's sake. |
| 876 | |
| 877 | Now that we've looked at the general extension syntax, we can return to the |
| 878 | features that simplify working with groups in complex REs. Since groups are |
| 879 | numbered from left to right and a complex expression may use many groups, it can |
| 880 | become difficult to keep track of the correct numbering. Modifying such a |
| 881 | complex RE is annoying, too: insert a new group near the beginning and you |
| 882 | change the numbers of everything that follows it. |
| 883 | |
| 884 | Sometimes you'll want to use a group to collect a part of a regular expression, |
| 885 | but aren't interested in retrieving the group's contents. You can make this fact |
| 886 | explicit by using a non-capturing group: ``(?:...)``, where you can replace the |
| 887 | ``...`` with any other regular expression. :: |
| 888 | |
| 889 | >>> m = re.match("([abc])+", "abc") |
| 890 | >>> m.groups() |
| 891 | ('c',) |
| 892 | >>> m = re.match("(?:[abc])+", "abc") |
| 893 | >>> m.groups() |
| 894 | () |
| 895 | |
| 896 | Except for the fact that you can't retrieve the contents of what the group |
| 897 | matched, a non-capturing group behaves exactly the same as a capturing group; |
| 898 | you can put anything inside it, repeat it with a repetition metacharacter such |
| 899 | as ``*``, and nest it within other groups (capturing or non-capturing). |
| 900 | ``(?:...)`` is particularly useful when modifying an existing pattern, since you |
| 901 | can add new groups without changing how all the other groups are numbered. It |
| 902 | should be mentioned that there's no performance difference in searching between |
| 903 | capturing and non-capturing groups; neither form is any faster than the other. |
| 904 | |
| 905 | A more significant feature is named groups: instead of referring to them by |
| 906 | numbers, groups can be referenced by a name. |
| 907 | |
| 908 | The syntax for a named group is one of the Python-specific extensions: |
| 909 | ``(?P<name>...)``. *name* is, obviously, the name of the group. Named groups |
| 910 | also behave exactly like capturing groups, and additionally associate a name |
| 911 | with a group. The :class:`MatchObject` methods that deal with capturing groups |
| 912 | all accept either integers that refer to the group by number or strings that |
| 913 | contain the desired group's name. Named groups are still given numbers, so you |
| 914 | can retrieve information about a group in two ways:: |
| 915 | |
| 916 | >>> p = re.compile(r'(?P<word>\b\w+\b)') |
| 917 | >>> m = p.search( '(((( Lots of punctuation )))' ) |
| 918 | >>> m.group('word') |
| 919 | 'Lots' |
| 920 | >>> m.group(1) |
| 921 | 'Lots' |
| 922 | |
| 923 | Named groups are handy because they let you use easily-remembered names, instead |
| 924 | of having to remember numbers. Here's an example RE from the :mod:`imaplib` |
| 925 | module:: |
| 926 | |
| 927 | InternalDate = re.compile(r'INTERNALDATE "' |
| 928 | r'(?P<day>[ 123][0-9])-(?P<mon>[A-Z][a-z][a-z])-' |
Georg Brandl | a1c6a1c | 2009-01-03 21:26:05 +0000 | [diff] [blame] | 929 | r'(?P<year>[0-9][0-9][0-9][0-9])' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 930 | r' (?P<hour>[0-9][0-9]):(?P<min>[0-9][0-9]):(?P<sec>[0-9][0-9])' |
| 931 | r' (?P<zonen>[-+])(?P<zoneh>[0-9][0-9])(?P<zonem>[0-9][0-9])' |
| 932 | r'"') |
| 933 | |
| 934 | It's obviously much easier to retrieve ``m.group('zonem')``, instead of having |
| 935 | to remember to retrieve group 9. |
| 936 | |
| 937 | The syntax for backreferences in an expression such as ``(...)\1`` refers to the |
| 938 | number of the group. There's naturally a variant that uses the group name |
| 939 | instead of the number. This is another Python extension: ``(?P=name)`` indicates |
| 940 | that the contents of the group called *name* should again be matched at the |
| 941 | current point. The regular expression for finding doubled words, |
| 942 | ``(\b\w+)\s+\1`` can also be written as ``(?P<word>\b\w+)\s+(?P=word)``:: |
| 943 | |
| 944 | >>> p = re.compile(r'(?P<word>\b\w+)\s+(?P=word)') |
| 945 | >>> p.search('Paris in the the spring').group() |
| 946 | 'the the' |
| 947 | |
| 948 | |
| 949 | Lookahead Assertions |
| 950 | -------------------- |
| 951 | |
| 952 | Another zero-width assertion is the lookahead assertion. Lookahead assertions |
| 953 | are available in both positive and negative form, and look like this: |
| 954 | |
| 955 | ``(?=...)`` |
| 956 | Positive lookahead assertion. This succeeds if the contained regular |
| 957 | expression, represented here by ``...``, successfully matches at the current |
| 958 | location, and fails otherwise. But, once the contained expression has been |
| 959 | tried, the matching engine doesn't advance at all; the rest of the pattern is |
| 960 | tried right where the assertion started. |
| 961 | |
| 962 | ``(?!...)`` |
| 963 | Negative lookahead assertion. This is the opposite of the positive assertion; |
| 964 | it succeeds if the contained expression *doesn't* match at the current position |
| 965 | in the string. |
| 966 | |
| 967 | To make this concrete, let's look at a case where a lookahead is useful. |
| 968 | Consider a simple pattern to match a filename and split it apart into a base |
| 969 | name and an extension, separated by a ``.``. For example, in ``news.rc``, |
| 970 | ``news`` is the base name, and ``rc`` is the filename's extension. |
| 971 | |
| 972 | The pattern to match this is quite simple: |
| 973 | |
| 974 | ``.*[.].*$`` |
| 975 | |
| 976 | Notice that the ``.`` needs to be treated specially because it's a |
| 977 | metacharacter; I've put it inside a character class. Also notice the trailing |
| 978 | ``$``; this is added to ensure that all the rest of the string must be included |
| 979 | in the extension. This regular expression matches ``foo.bar`` and |
| 980 | ``autoexec.bat`` and ``sendmail.cf`` and ``printers.conf``. |
| 981 | |
| 982 | Now, consider complicating the problem a bit; what if you want to match |
| 983 | filenames where the extension is not ``bat``? Some incorrect attempts: |
| 984 | |
| 985 | ``.*[.][^b].*$`` The first attempt above tries to exclude ``bat`` by requiring |
| 986 | that the first character of the extension is not a ``b``. This is wrong, |
| 987 | because the pattern also doesn't match ``foo.bar``. |
| 988 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 989 | ``.*[.]([^b]..|.[^a].|..[^t])$`` |
| 990 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 991 | The expression gets messier when you try to patch up the first solution by |
| 992 | requiring one of the following cases to match: the first character of the |
| 993 | extension isn't ``b``; the second character isn't ``a``; or the third character |
| 994 | isn't ``t``. This accepts ``foo.bar`` and rejects ``autoexec.bat``, but it |
| 995 | requires a three-letter extension and won't accept a filename with a two-letter |
| 996 | extension such as ``sendmail.cf``. We'll complicate the pattern again in an |
| 997 | effort to fix it. |
| 998 | |
| 999 | ``.*[.]([^b].?.?|.[^a]?.?|..?[^t]?)$`` |
| 1000 | |
| 1001 | In the third attempt, the second and third letters are all made optional in |
| 1002 | order to allow matching extensions shorter than three characters, such as |
| 1003 | ``sendmail.cf``. |
| 1004 | |
| 1005 | The pattern's getting really complicated now, which makes it hard to read and |
| 1006 | understand. Worse, if the problem changes and you want to exclude both ``bat`` |
| 1007 | and ``exe`` as extensions, the pattern would get even more complicated and |
| 1008 | confusing. |
| 1009 | |
| 1010 | A negative lookahead cuts through all this confusion: |
| 1011 | |
| 1012 | ``.*[.](?!bat$).*$`` The negative lookahead means: if the expression ``bat`` |
| 1013 | doesn't match at this point, try the rest of the pattern; if ``bat$`` does |
| 1014 | match, the whole pattern will fail. The trailing ``$`` is required to ensure |
| 1015 | that something like ``sample.batch``, where the extension only starts with |
| 1016 | ``bat``, will be allowed. |
| 1017 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1018 | Excluding another filename extension is now easy; simply add it as an |
| 1019 | alternative inside the assertion. The following pattern excludes filenames that |
| 1020 | end in either ``bat`` or ``exe``: |
| 1021 | |
| 1022 | ``.*[.](?!bat$|exe$).*$`` |
| 1023 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1024 | |
| 1025 | Modifying Strings |
| 1026 | ================= |
| 1027 | |
| 1028 | Up to this point, we've simply performed searches against a static string. |
| 1029 | Regular expressions are also commonly used to modify strings in various ways, |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 1030 | using the following pattern methods: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1031 | |
| 1032 | +------------------+-----------------------------------------------+ |
| 1033 | | Method/Attribute | Purpose | |
| 1034 | +==================+===============================================+ |
| 1035 | | ``split()`` | Split the string into a list, splitting it | |
| 1036 | | | wherever the RE matches | |
| 1037 | +------------------+-----------------------------------------------+ |
| 1038 | | ``sub()`` | Find all substrings where the RE matches, and | |
| 1039 | | | replace them with a different string | |
| 1040 | +------------------+-----------------------------------------------+ |
| 1041 | | ``subn()`` | Does the same thing as :meth:`sub`, but | |
| 1042 | | | returns the new string and the number of | |
| 1043 | | | replacements | |
| 1044 | +------------------+-----------------------------------------------+ |
| 1045 | |
| 1046 | |
| 1047 | Splitting Strings |
| 1048 | ----------------- |
| 1049 | |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 1050 | The :meth:`split` method of a pattern splits a string apart |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1051 | wherever the RE matches, returning a list of the pieces. It's similar to the |
| 1052 | :meth:`split` method of strings but provides much more generality in the |
| 1053 | delimiters that you can split by; :meth:`split` only supports splitting by |
| 1054 | whitespace or by a fixed string. As you'd expect, there's a module-level |
| 1055 | :func:`re.split` function, too. |
| 1056 | |
| 1057 | |
| 1058 | .. method:: .split(string [, maxsplit=0]) |
| 1059 | :noindex: |
| 1060 | |
| 1061 | Split *string* by the matches of the regular expression. If capturing |
| 1062 | parentheses are used in the RE, then their contents will also be returned as |
| 1063 | part of the resulting list. If *maxsplit* is nonzero, at most *maxsplit* splits |
| 1064 | are performed. |
| 1065 | |
| 1066 | You can limit the number of splits made, by passing a value for *maxsplit*. |
| 1067 | When *maxsplit* is nonzero, at most *maxsplit* splits will be made, and the |
| 1068 | remainder of the string is returned as the final element of the list. In the |
| 1069 | following example, the delimiter is any sequence of non-alphanumeric characters. |
| 1070 | :: |
| 1071 | |
| 1072 | >>> p = re.compile(r'\W+') |
| 1073 | >>> p.split('This is a test, short and sweet, of split().') |
| 1074 | ['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', ''] |
| 1075 | >>> p.split('This is a test, short and sweet, of split().', 3) |
| 1076 | ['This', 'is', 'a', 'test, short and sweet, of split().'] |
| 1077 | |
| 1078 | Sometimes you're not only interested in what the text between delimiters is, but |
| 1079 | also need to know what the delimiter was. If capturing parentheses are used in |
| 1080 | the RE, then their values are also returned as part of the list. Compare the |
| 1081 | following calls:: |
| 1082 | |
| 1083 | >>> p = re.compile(r'\W+') |
| 1084 | >>> p2 = re.compile(r'(\W+)') |
| 1085 | >>> p.split('This... is a test.') |
| 1086 | ['This', 'is', 'a', 'test', ''] |
| 1087 | >>> p2.split('This... is a test.') |
| 1088 | ['This', '... ', 'is', ' ', 'a', ' ', 'test', '.', ''] |
| 1089 | |
| 1090 | The module-level function :func:`re.split` adds the RE to be used as the first |
| 1091 | argument, but is otherwise the same. :: |
| 1092 | |
| 1093 | >>> re.split('[\W]+', 'Words, words, words.') |
| 1094 | ['Words', 'words', 'words', ''] |
| 1095 | >>> re.split('([\W]+)', 'Words, words, words.') |
| 1096 | ['Words', ', ', 'words', ', ', 'words', '.', ''] |
| 1097 | >>> re.split('[\W]+', 'Words, words, words.', 1) |
| 1098 | ['Words', 'words, words.'] |
| 1099 | |
| 1100 | |
| 1101 | Search and Replace |
| 1102 | ------------------ |
| 1103 | |
| 1104 | Another common task is to find all the matches for a pattern, and replace them |
| 1105 | with a different string. The :meth:`sub` method takes a replacement value, |
| 1106 | which can be either a string or a function, and the string to be processed. |
| 1107 | |
| 1108 | |
| 1109 | .. method:: .sub(replacement, string[, count=0]) |
| 1110 | :noindex: |
| 1111 | |
| 1112 | Returns the string obtained by replacing the leftmost non-overlapping |
| 1113 | occurrences of the RE in *string* by the replacement *replacement*. If the |
| 1114 | pattern isn't found, *string* is returned unchanged. |
| 1115 | |
| 1116 | The optional argument *count* is the maximum number of pattern occurrences to be |
| 1117 | replaced; *count* must be a non-negative integer. The default value of 0 means |
| 1118 | to replace all occurrences. |
| 1119 | |
| 1120 | Here's a simple example of using the :meth:`sub` method. It replaces colour |
| 1121 | names with the word ``colour``:: |
| 1122 | |
| 1123 | >>> p = re.compile( '(blue|white|red)') |
| 1124 | >>> p.sub( 'colour', 'blue socks and red shoes') |
| 1125 | 'colour socks and colour shoes' |
| 1126 | >>> p.sub( 'colour', 'blue socks and red shoes', count=1) |
| 1127 | 'colour socks and red shoes' |
| 1128 | |
| 1129 | The :meth:`subn` method does the same work, but returns a 2-tuple containing the |
| 1130 | new string value and the number of replacements that were performed:: |
| 1131 | |
| 1132 | >>> p = re.compile( '(blue|white|red)') |
| 1133 | >>> p.subn( 'colour', 'blue socks and red shoes') |
| 1134 | ('colour socks and colour shoes', 2) |
| 1135 | >>> p.subn( 'colour', 'no colours at all') |
| 1136 | ('no colours at all', 0) |
| 1137 | |
| 1138 | Empty matches are replaced only when they're not adjacent to a previous match. |
| 1139 | :: |
| 1140 | |
| 1141 | >>> p = re.compile('x*') |
| 1142 | >>> p.sub('-', 'abxd') |
| 1143 | '-a-b-d-' |
| 1144 | |
| 1145 | If *replacement* is a string, any backslash escapes in it are processed. That |
| 1146 | is, ``\n`` is converted to a single newline character, ``\r`` is converted to a |
| 1147 | carriage return, and so forth. Unknown escapes such as ``\j`` are left alone. |
| 1148 | Backreferences, such as ``\6``, are replaced with the substring matched by the |
| 1149 | corresponding group in the RE. This lets you incorporate portions of the |
| 1150 | original text in the resulting replacement string. |
| 1151 | |
| 1152 | This example matches the word ``section`` followed by a string enclosed in |
| 1153 | ``{``, ``}``, and changes ``section`` to ``subsection``:: |
| 1154 | |
| 1155 | >>> p = re.compile('section{ ( [^}]* ) }', re.VERBOSE) |
| 1156 | >>> p.sub(r'subsection{\1}','section{First} section{second}') |
| 1157 | 'subsection{First} subsection{second}' |
| 1158 | |
| 1159 | There's also a syntax for referring to named groups as defined by the |
| 1160 | ``(?P<name>...)`` syntax. ``\g<name>`` will use the substring matched by the |
| 1161 | group named ``name``, and ``\g<number>`` uses the corresponding group number. |
| 1162 | ``\g<2>`` is therefore equivalent to ``\2``, but isn't ambiguous in a |
| 1163 | replacement string such as ``\g<2>0``. (``\20`` would be interpreted as a |
| 1164 | reference to group 20, not a reference to group 2 followed by the literal |
| 1165 | character ``'0'``.) The following substitutions are all equivalent, but use all |
| 1166 | three variations of the replacement string. :: |
| 1167 | |
| 1168 | >>> p = re.compile('section{ (?P<name> [^}]* ) }', re.VERBOSE) |
| 1169 | >>> p.sub(r'subsection{\1}','section{First}') |
| 1170 | 'subsection{First}' |
| 1171 | >>> p.sub(r'subsection{\g<1>}','section{First}') |
| 1172 | 'subsection{First}' |
| 1173 | >>> p.sub(r'subsection{\g<name>}','section{First}') |
| 1174 | 'subsection{First}' |
| 1175 | |
| 1176 | *replacement* can also be a function, which gives you even more control. If |
| 1177 | *replacement* is a function, the function is called for every non-overlapping |
| 1178 | occurrence of *pattern*. On each call, the function is passed a |
| 1179 | :class:`MatchObject` argument for the match and can use this information to |
| 1180 | compute the desired replacement string and return it. |
| 1181 | |
| 1182 | In the following example, the replacement function translates decimals into |
| 1183 | hexadecimal:: |
| 1184 | |
| 1185 | >>> def hexrepl( match ): |
| 1186 | ... "Return the hex string for a decimal number" |
| 1187 | ... value = int( match.group() ) |
| 1188 | ... return hex(value) |
| 1189 | ... |
| 1190 | >>> p = re.compile(r'\d+') |
| 1191 | >>> p.sub(hexrepl, 'Call 65490 for printing, 49152 for user code.') |
| 1192 | 'Call 0xffd2 for printing, 0xc000 for user code.' |
| 1193 | |
| 1194 | When using the module-level :func:`re.sub` function, the pattern is passed as |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 1195 | the first argument. The pattern may be provided as an object or as a string; if |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1196 | you need to specify regular expression flags, you must either use a |
Benjamin Peterson | 8cc7d88 | 2009-06-01 23:14:51 +0000 | [diff] [blame] | 1197 | pattern object as the first parameter, or use embedded modifiers in the |
| 1198 | pattern string, e.g. ``sub("(?i)b+", "x", "bbbb BBBB")`` returns ``'x x'``. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1199 | |
| 1200 | |
| 1201 | Common Problems |
| 1202 | =============== |
| 1203 | |
| 1204 | Regular expressions are a powerful tool for some applications, but in some ways |
| 1205 | their behaviour isn't intuitive and at times they don't behave the way you may |
| 1206 | expect them to. This section will point out some of the most common pitfalls. |
| 1207 | |
| 1208 | |
| 1209 | Use String Methods |
| 1210 | ------------------ |
| 1211 | |
| 1212 | Sometimes using the :mod:`re` module is a mistake. If you're matching a fixed |
| 1213 | string, or a single character class, and you're not using any :mod:`re` features |
| 1214 | such as the :const:`IGNORECASE` flag, then the full power of regular expressions |
| 1215 | may not be required. Strings have several methods for performing operations with |
| 1216 | fixed strings and they're usually much faster, because the implementation is a |
| 1217 | single small C loop that's been optimized for the purpose, instead of the large, |
| 1218 | more generalized regular expression engine. |
| 1219 | |
| 1220 | One example might be replacing a single fixed string with another one; for |
| 1221 | example, you might replace ``word`` with ``deed``. ``re.sub()`` seems like the |
| 1222 | function to use for this, but consider the :meth:`replace` method. Note that |
| 1223 | :func:`replace` will also replace ``word`` inside words, turning ``swordfish`` |
| 1224 | into ``sdeedfish``, but the naive RE ``word`` would have done that, too. (To |
| 1225 | avoid performing the substitution on parts of words, the pattern would have to |
| 1226 | be ``\bword\b``, in order to require that ``word`` have a word boundary on |
| 1227 | either side. This takes the job beyond :meth:`replace`'s abilities.) |
| 1228 | |
| 1229 | Another common task is deleting every occurrence of a single character from a |
| 1230 | string or replacing it with another single character. You might do this with |
| 1231 | something like ``re.sub('\n', ' ', S)``, but :meth:`translate` is capable of |
| 1232 | doing both tasks and will be faster than any regular expression operation can |
| 1233 | be. |
| 1234 | |
| 1235 | In short, before turning to the :mod:`re` module, consider whether your problem |
| 1236 | can be solved with a faster and simpler string method. |
| 1237 | |
| 1238 | |
| 1239 | match() versus search() |
| 1240 | ----------------------- |
| 1241 | |
| 1242 | The :func:`match` function only checks if the RE matches at the beginning of the |
| 1243 | string while :func:`search` will scan forward through the string for a match. |
| 1244 | It's important to keep this distinction in mind. Remember, :func:`match` will |
| 1245 | only report a successful match which will start at 0; if the match wouldn't |
| 1246 | start at zero, :func:`match` will *not* report it. :: |
| 1247 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1248 | >>> print(re.match('super', 'superstition').span()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1249 | (0, 5) |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1250 | >>> print(re.match('super', 'insuperable')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1251 | None |
| 1252 | |
| 1253 | On the other hand, :func:`search` will scan forward through the string, |
| 1254 | reporting the first match it finds. :: |
| 1255 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1256 | >>> print(re.search('super', 'superstition').span()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1257 | (0, 5) |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1258 | >>> print(re.search('super', 'insuperable').span()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1259 | (2, 7) |
| 1260 | |
| 1261 | Sometimes you'll be tempted to keep using :func:`re.match`, and just add ``.*`` |
| 1262 | to the front of your RE. Resist this temptation and use :func:`re.search` |
| 1263 | instead. The regular expression compiler does some analysis of REs in order to |
| 1264 | speed up the process of looking for a match. One such analysis figures out what |
| 1265 | the first character of a match must be; for example, a pattern starting with |
| 1266 | ``Crow`` must match starting with a ``'C'``. The analysis lets the engine |
| 1267 | quickly scan through the string looking for the starting character, only trying |
| 1268 | the full match if a ``'C'`` is found. |
| 1269 | |
| 1270 | Adding ``.*`` defeats this optimization, requiring scanning to the end of the |
| 1271 | string and then backtracking to find a match for the rest of the RE. Use |
| 1272 | :func:`re.search` instead. |
| 1273 | |
| 1274 | |
| 1275 | Greedy versus Non-Greedy |
| 1276 | ------------------------ |
| 1277 | |
| 1278 | When repeating a regular expression, as in ``a*``, the resulting action is to |
| 1279 | consume as much of the pattern as possible. This fact often bites you when |
| 1280 | you're trying to match a pair of balanced delimiters, such as the angle brackets |
| 1281 | surrounding an HTML tag. The naive pattern for matching a single HTML tag |
| 1282 | doesn't work because of the greedy nature of ``.*``. :: |
| 1283 | |
| 1284 | >>> s = '<html><head><title>Title</title>' |
| 1285 | >>> len(s) |
| 1286 | 32 |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1287 | >>> print(re.match('<.*>', s).span()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1288 | (0, 32) |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1289 | >>> print(re.match('<.*>', s).group()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1290 | <html><head><title>Title</title> |
| 1291 | |
| 1292 | The RE matches the ``'<'`` in ``<html>``, and the ``.*`` consumes the rest of |
| 1293 | the string. There's still more left in the RE, though, and the ``>`` can't |
| 1294 | match at the end of the string, so the regular expression engine has to |
| 1295 | backtrack character by character until it finds a match for the ``>``. The |
| 1296 | final match extends from the ``'<'`` in ``<html>`` to the ``'>'`` in |
| 1297 | ``</title>``, which isn't what you want. |
| 1298 | |
| 1299 | In this case, the solution is to use the non-greedy qualifiers ``*?``, ``+?``, |
| 1300 | ``??``, or ``{m,n}?``, which match as *little* text as possible. In the above |
| 1301 | example, the ``'>'`` is tried immediately after the first ``'<'`` matches, and |
| 1302 | when it fails, the engine advances a character at a time, retrying the ``'>'`` |
| 1303 | at every step. This produces just the right result:: |
| 1304 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 1305 | >>> print(re.match('<.*?>', s).group()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1306 | <html> |
| 1307 | |
| 1308 | (Note that parsing HTML or XML with regular expressions is painful. |
| 1309 | Quick-and-dirty patterns will handle common cases, but HTML and XML have special |
| 1310 | cases that will break the obvious regular expression; by the time you've written |
| 1311 | a regular expression that handles all of the possible cases, the patterns will |
| 1312 | be *very* complicated. Use an HTML or XML parser module for such tasks.) |
| 1313 | |
| 1314 | |
Terry Reedy | 8663e34 | 2011-01-10 21:49:11 +0000 | [diff] [blame] | 1315 | Using re.VERBOSE |
| 1316 | ---------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1317 | |
| 1318 | By now you've probably noticed that regular expressions are a very compact |
| 1319 | notation, but they're not terribly readable. REs of moderate complexity can |
| 1320 | become lengthy collections of backslashes, parentheses, and metacharacters, |
| 1321 | making them difficult to read and understand. |
| 1322 | |
| 1323 | For such REs, specifying the ``re.VERBOSE`` flag when compiling the regular |
| 1324 | expression can be helpful, because it allows you to format the regular |
| 1325 | expression more clearly. |
| 1326 | |
| 1327 | The ``re.VERBOSE`` flag has several effects. Whitespace in the regular |
| 1328 | expression that *isn't* inside a character class is ignored. This means that an |
| 1329 | expression such as ``dog | cat`` is equivalent to the less readable ``dog|cat``, |
| 1330 | but ``[a b]`` will still match the characters ``'a'``, ``'b'``, or a space. In |
| 1331 | addition, you can also put comments inside a RE; comments extend from a ``#`` |
| 1332 | character to the next newline. When used with triple-quoted strings, this |
| 1333 | enables REs to be formatted more neatly:: |
| 1334 | |
| 1335 | pat = re.compile(r""" |
| 1336 | \s* # Skip leading whitespace |
| 1337 | (?P<header>[^:]+) # Header name |
| 1338 | \s* : # Whitespace, and a colon |
| 1339 | (?P<value>.*?) # The header's value -- *? used to |
| 1340 | # lose the following trailing whitespace |
| 1341 | \s*$ # Trailing whitespace to end-of-line |
| 1342 | """, re.VERBOSE) |
| 1343 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 1344 | This is far more readable than:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1345 | |
| 1346 | pat = re.compile(r"\s*(?P<header>[^:]+)\s*:(?P<value>.*?)\s*$") |
| 1347 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1348 | |
| 1349 | Feedback |
| 1350 | ======== |
| 1351 | |
| 1352 | Regular expressions are a complicated topic. Did this document help you |
| 1353 | understand them? Were there parts that were unclear, or Problems you |
| 1354 | encountered that weren't covered here? If so, please send suggestions for |
| 1355 | improvements to the author. |
| 1356 | |
| 1357 | The most complete book on regular expressions is almost certainly Jeffrey |
| 1358 | Friedl's Mastering Regular Expressions, published by O'Reilly. Unfortunately, |
| 1359 | it exclusively concentrates on Perl and Java's flavours of regular expressions, |
| 1360 | and doesn't contain any Python material at all, so it won't be useful as a |
| 1361 | reference for programming in Python. (The first edition covered Python's |
| 1362 | now-removed :mod:`regex` module, which won't help you much.) Consider checking |
| 1363 | it out from your library. |
| 1364 | |