Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 1 | # |
| 2 | # Secret Labs' Regular Expression Engine |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 3 | # |
| 4 | # re-compatible interface for the sre matching engine |
| 5 | # |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 6 | # Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved. |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 7 | # |
Fredrik Lundh | 29c4ba9 | 2000-08-01 18:20:07 +0000 | [diff] [blame] | 8 | # This version of the SRE library can be redistributed under CNRI's |
| 9 | # Python 1.6 license. For any other use, please contact Secret Labs |
| 10 | # AB (info@pythonware.com). |
| 11 | # |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 12 | # Portions of this engine have been developed in cooperation with |
Fredrik Lundh | 29c4ba9 | 2000-08-01 18:20:07 +0000 | [diff] [blame] | 13 | # CNRI. Hewlett-Packard provided funding for 1.6 integration and |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 14 | # other compatibility work. |
| 15 | # |
| 16 | |
Fred Drake | 9f5b822 | 2001-09-04 19:20:06 +0000 | [diff] [blame] | 17 | r"""Support for regular expressions (RE). |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 18 | |
| 19 | This module provides regular expression matching operations similar to |
Fredrik Lundh | dac5849 | 2001-10-21 21:48:30 +0000 | [diff] [blame] | 20 | those found in Perl. It supports both 8-bit and Unicode strings; both |
| 21 | the pattern and the strings being processed can contain null bytes and |
| 22 | characters outside the US ASCII range. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 23 | |
Fredrik Lundh | dac5849 | 2001-10-21 21:48:30 +0000 | [diff] [blame] | 24 | Regular expressions can contain both special and ordinary characters. |
| 25 | Most ordinary characters, like "A", "a", or "0", are the simplest |
| 26 | regular expressions; they simply match themselves. You can |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 27 | concatenate ordinary characters, so last matches the string 'last'. |
| 28 | |
| 29 | The special characters are: |
| 30 | "." Matches any character except a newline. |
| 31 | "^" Matches the start of the string. |
Georg Brandl | 86def6c | 2008-01-21 20:36:10 +0000 | [diff] [blame] | 32 | "$" Matches the end of the string or just before the newline at |
| 33 | the end of the string. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 34 | "*" Matches 0 or more (greedy) repetitions of the preceding RE. |
| 35 | Greedy means that it will match as many repetitions as possible. |
| 36 | "+" Matches 1 or more (greedy) repetitions of the preceding RE. |
| 37 | "?" Matches 0 or 1 (greedy) of the preceding RE. |
| 38 | *?,+?,?? Non-greedy versions of the previous three special characters. |
| 39 | {m,n} Matches from m to n repetitions of the preceding RE. |
| 40 | {m,n}? Non-greedy version of the above. |
Christian Heimes | af98da1 | 2008-01-27 15:18:18 +0000 | [diff] [blame] | 41 | "\\" Either escapes special characters or signals a special sequence. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 42 | [] Indicates a set of characters. |
| 43 | A "^" as the first character indicates a complementing set. |
| 44 | "|" A|B, creates an RE that will match either A or B. |
| 45 | (...) Matches the RE inside the parentheses. |
| 46 | The contents can be retrieved or matched later in the string. |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 47 | (?aiLmsux) Set the A, I, L, M, S, U, or X flag for the RE (see below). |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 48 | (?:...) Non-grouping version of regular parentheses. |
| 49 | (?P<name>...) The substring matched by the group is accessible by name. |
| 50 | (?P=name) Matches the text matched earlier by the group named name. |
| 51 | (?#...) A comment; ignored. |
| 52 | (?=...) Matches if ... matches next, but doesn't consume the string. |
| 53 | (?!...) Matches if ... doesn't match next. |
Christian Heimes | af98da1 | 2008-01-27 15:18:18 +0000 | [diff] [blame] | 54 | (?<=...) Matches if preceded by ... (must be fixed length). |
| 55 | (?<!...) Matches if not preceded by ... (must be fixed length). |
| 56 | (?(id/name)yes|no) Matches yes pattern if the group with id/name matched, |
| 57 | the (optional) no pattern otherwise. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 58 | |
| 59 | The special sequences consist of "\\" and a character from the list |
Fredrik Lundh | dac5849 | 2001-10-21 21:48:30 +0000 | [diff] [blame] | 60 | below. If the ordinary character is not on the list, then the |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 61 | resulting RE will match the second character. |
Fred Drake | 9f5b822 | 2001-09-04 19:20:06 +0000 | [diff] [blame] | 62 | \number Matches the contents of the group of the same number. |
| 63 | \A Matches only at the start of the string. |
| 64 | \Z Matches only at the end of the string. |
| 65 | \b Matches the empty string, but only at the start or end of a word. |
| 66 | \B Matches the empty string, but not at the start or end of a word. |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 67 | \d Matches any decimal digit; equivalent to the set [0-9] in |
| 68 | bytes patterns or string patterns with the ASCII flag. |
| 69 | In string patterns without the ASCII flag, it will match the whole |
| 70 | range of Unicode digits. |
| 71 | \D Matches any non-digit character; equivalent to [^\d]. |
Fred Drake | 9f5b822 | 2001-09-04 19:20:06 +0000 | [diff] [blame] | 72 | \s Matches any whitespace character; equivalent to [ \t\n\r\f\v]. |
| 73 | \S Matches any non-whitespace character; equiv. to [^ \t\n\r\f\v]. |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 74 | \w Matches any alphanumeric character; equivalent to [a-zA-Z0-9_] |
| 75 | in bytes patterns or string patterns with the ASCII flag. |
| 76 | In string patterns without the ASCII flag, it will match the |
| 77 | range of Unicode alphanumeric characters (letters plus digits |
| 78 | plus underscore). |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 79 | With LOCALE, it will match the set [0-9_] plus characters defined |
| 80 | as letters for the current locale. |
Fred Drake | 9f5b822 | 2001-09-04 19:20:06 +0000 | [diff] [blame] | 81 | \W Matches the complement of \w. |
| 82 | \\ Matches a literal backslash. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 83 | |
| 84 | This module exports the following functions: |
| 85 | match Match a regular expression pattern to the beginning of a string. |
| 86 | search Search a string for the presence of a pattern. |
| 87 | sub Substitute occurrences of a pattern found in a string. |
| 88 | subn Same as sub, but also return the number of substitutions made. |
| 89 | split Split a string by the occurrences of a pattern. |
| 90 | findall Find all occurrences of a pattern in a string. |
Christian Heimes | af98da1 | 2008-01-27 15:18:18 +0000 | [diff] [blame] | 91 | finditer Return an iterator yielding a match object for each match. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 92 | compile Compile a pattern into a RegexObject. |
| 93 | purge Clear the regular expression cache. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 94 | escape Backslash all non-alphanumerics in a string. |
| 95 | |
| 96 | Some of the functions in this module takes flags as optional parameters: |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 97 | A ASCII For string patterns, make \w, \W, \b, \B, \d, \D |
| 98 | match the corresponding ASCII character categories |
| 99 | (rather than the whole Unicode categories, which is the |
| 100 | default). |
| 101 | For bytes patterns, this flag is the only available |
| 102 | behaviour and needn't be specified. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 103 | I IGNORECASE Perform case-insensitive matching. |
| 104 | L LOCALE Make \w, \W, \b, \B, dependent on the current locale. |
Georg Brandl | 86def6c | 2008-01-21 20:36:10 +0000 | [diff] [blame] | 105 | M MULTILINE "^" matches the beginning of lines (after a newline) |
| 106 | as well as the string. |
| 107 | "$" matches the end of lines (before a newline) as well |
| 108 | as the end of the string. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 109 | S DOTALL "." matches any character at all, including the newline. |
| 110 | X VERBOSE Ignore whitespace and comments for nicer looking RE's. |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 111 | U UNICODE For compatibility only. Ignored for string patterns (it |
| 112 | is the default), and forbidden for bytes patterns. |
Fred Drake | b8f2274 | 2001-09-04 19:10:20 +0000 | [diff] [blame] | 113 | |
| 114 | This module also defines an exception 'error'. |
| 115 | |
| 116 | """ |
Fredrik Lundh | 21009b9 | 2001-09-18 18:47:09 +0000 | [diff] [blame] | 117 | |
Fredrik Lundh | 703ce81 | 2001-10-24 22:16:30 +0000 | [diff] [blame] | 118 | import sys |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 119 | import sre_compile |
Fredrik Lundh | 436c3d58 | 2000-06-29 08:58:44 +0000 | [diff] [blame] | 120 | import sre_parse |
Antoine Pitrou | 3060c45 | 2010-08-13 16:27:38 +0000 | [diff] [blame] | 121 | import functools |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 122 | |
Fredrik Lundh | f2989b2 | 2001-02-18 12:05:16 +0000 | [diff] [blame] | 123 | # public symbols |
| 124 | __all__ = [ "match", "search", "sub", "subn", "split", "findall", |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 125 | "compile", "purge", "template", "escape", "A", "I", "L", "M", "S", "X", |
| 126 | "U", "ASCII", "IGNORECASE", "LOCALE", "MULTILINE", "DOTALL", "VERBOSE", |
Fredrik Lundh | f2989b2 | 2001-02-18 12:05:16 +0000 | [diff] [blame] | 127 | "UNICODE", "error" ] |
| 128 | |
Fredrik Lundh | bec95b9 | 2001-10-21 16:47:57 +0000 | [diff] [blame] | 129 | __version__ = "2.2.1" |
Fredrik Lundh | b25e1ad | 2001-03-22 15:50:10 +0000 | [diff] [blame] | 130 | |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 131 | # flags |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 132 | A = ASCII = sre_compile.SRE_FLAG_ASCII # assume ascii "locale" |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 133 | I = IGNORECASE = sre_compile.SRE_FLAG_IGNORECASE # ignore case |
| 134 | L = LOCALE = sre_compile.SRE_FLAG_LOCALE # assume current 8-bit locale |
Antoine Pitrou | fd03645 | 2008-08-19 17:56:33 +0000 | [diff] [blame] | 135 | U = UNICODE = sre_compile.SRE_FLAG_UNICODE # assume unicode "locale" |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 136 | M = MULTILINE = sre_compile.SRE_FLAG_MULTILINE # make anchors look for newline |
| 137 | S = DOTALL = sre_compile.SRE_FLAG_DOTALL # make dot match newline |
| 138 | X = VERBOSE = sre_compile.SRE_FLAG_VERBOSE # ignore whitespace and comments |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 139 | |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 140 | # sre extensions (experimental, don't rely on these) |
| 141 | T = TEMPLATE = sre_compile.SRE_FLAG_TEMPLATE # disable backtracking |
| 142 | DEBUG = sre_compile.SRE_FLAG_DEBUG # dump pattern after compilation |
Fredrik Lundh | 436c3d58 | 2000-06-29 08:58:44 +0000 | [diff] [blame] | 143 | |
| 144 | # sre exception |
Fredrik Lundh | be2211e | 2000-06-29 16:57:40 +0000 | [diff] [blame] | 145 | error = sre_compile.error |
Fredrik Lundh | 436c3d58 | 2000-06-29 08:58:44 +0000 | [diff] [blame] | 146 | |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 147 | # -------------------------------------------------------------------- |
| 148 | # public interface |
| 149 | |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 150 | def match(pattern, string, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 151 | """Try to apply the pattern at the start of the string, returning |
| 152 | a match object, or None if no match was found.""" |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 153 | return _compile(pattern, flags).match(string) |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 154 | |
| 155 | def search(pattern, string, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 156 | """Scan through string looking for a match to the pattern, returning |
| 157 | a match object, or None if no match was found.""" |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 158 | return _compile(pattern, flags).search(string) |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 159 | |
Gregory P. Smith | ccc5ae7 | 2009-03-02 05:21:55 +0000 | [diff] [blame] | 160 | def sub(pattern, repl, string, count=0, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 161 | """Return the string obtained by replacing the leftmost |
| 162 | non-overlapping occurrences of the pattern in string by the |
Andrew M. Kuchling | 43ab0cd | 2004-08-07 17:41:54 +0000 | [diff] [blame] | 163 | replacement repl. repl can be either a string or a callable; |
Benjamin Peterson | 75edad0 | 2009-01-01 15:05:06 +0000 | [diff] [blame] | 164 | if a string, backslash escapes in it are processed. If it is |
| 165 | a callable, it's passed the match object and must return |
Andrew M. Kuchling | 43ab0cd | 2004-08-07 17:41:54 +0000 | [diff] [blame] | 166 | a replacement string to be used.""" |
Gregory P. Smith | ccc5ae7 | 2009-03-02 05:21:55 +0000 | [diff] [blame] | 167 | return _compile(pattern, flags).sub(repl, string, count) |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 168 | |
Gregory P. Smith | ccc5ae7 | 2009-03-02 05:21:55 +0000 | [diff] [blame] | 169 | def subn(pattern, repl, string, count=0, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 170 | """Return a 2-tuple containing (new_string, number). |
| 171 | new_string is the string obtained by replacing the leftmost |
| 172 | non-overlapping occurrences of the pattern in the source |
| 173 | string by the replacement repl. number is the number of |
Andrew M. Kuchling | 43ab0cd | 2004-08-07 17:41:54 +0000 | [diff] [blame] | 174 | substitutions that were made. repl can be either a string or a |
Benjamin Peterson | 75edad0 | 2009-01-01 15:05:06 +0000 | [diff] [blame] | 175 | callable; if a string, backslash escapes in it are processed. |
| 176 | If it is a callable, it's passed the match object and must |
Andrew M. Kuchling | 43ab0cd | 2004-08-07 17:41:54 +0000 | [diff] [blame] | 177 | return a replacement string to be used.""" |
Gregory P. Smith | ccc5ae7 | 2009-03-02 05:21:55 +0000 | [diff] [blame] | 178 | return _compile(pattern, flags).subn(repl, string, count) |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 179 | |
Gregory P. Smith | ccc5ae7 | 2009-03-02 05:21:55 +0000 | [diff] [blame] | 180 | def split(pattern, string, maxsplit=0, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 181 | """Split the source string by the occurrences of the pattern, |
| 182 | returning a list containing the resulting substrings.""" |
Gregory P. Smith | ccc5ae7 | 2009-03-02 05:21:55 +0000 | [diff] [blame] | 183 | return _compile(pattern, flags).split(string, maxsplit) |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 184 | |
Raymond Hettinger | 596ba4d | 2004-09-24 03:41:05 +0000 | [diff] [blame] | 185 | def findall(pattern, string, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 186 | """Return a list of all non-overlapping matches in the string. |
| 187 | |
| 188 | If one or more groups are present in the pattern, return a |
| 189 | list of groups; this will be a list of tuples if the pattern |
| 190 | has more than one group. |
| 191 | |
| 192 | Empty matches are included in the result.""" |
Raymond Hettinger | 596ba4d | 2004-09-24 03:41:05 +0000 | [diff] [blame] | 193 | return _compile(pattern, flags).findall(string) |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 194 | |
Fredrik Lundh | 703ce81 | 2001-10-24 22:16:30 +0000 | [diff] [blame] | 195 | if sys.hexversion >= 0x02020000: |
Guido van Rossum | 0d97655 | 2002-10-14 12:22:17 +0000 | [diff] [blame] | 196 | __all__.append("finditer") |
Raymond Hettinger | 596ba4d | 2004-09-24 03:41:05 +0000 | [diff] [blame] | 197 | def finditer(pattern, string, flags=0): |
Fredrik Lundh | b7747e2 | 2001-10-28 20:15:40 +0000 | [diff] [blame] | 198 | """Return an iterator over all non-overlapping matches in the |
| 199 | string. For each match, the iterator returns a match object. |
Fredrik Lundh | 703ce81 | 2001-10-24 22:16:30 +0000 | [diff] [blame] | 200 | |
| 201 | Empty matches are included in the result.""" |
Raymond Hettinger | 596ba4d | 2004-09-24 03:41:05 +0000 | [diff] [blame] | 202 | return _compile(pattern, flags).finditer(string) |
Fredrik Lundh | 703ce81 | 2001-10-24 22:16:30 +0000 | [diff] [blame] | 203 | |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 204 | def compile(pattern, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 205 | "Compile a regular expression pattern, returning a pattern object." |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 206 | return _compile(pattern, flags) |
| 207 | |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 208 | def purge(): |
Antoine Pitrou | 3060c45 | 2010-08-13 16:27:38 +0000 | [diff] [blame] | 209 | "Clear the regular expression caches" |
Raymond Hettinger | 02566ec | 2010-09-04 22:46:06 +0000 | [diff] [blame] | 210 | _compile_typed.cache_clear() |
| 211 | _compile_repl.cache_clear() |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 212 | |
Fredrik Lundh | 436c3d58 | 2000-06-29 08:58:44 +0000 | [diff] [blame] | 213 | def template(pattern, flags=0): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 214 | "Compile a template pattern, returning a pattern object" |
Fredrik Lundh | 436c3d58 | 2000-06-29 08:58:44 +0000 | [diff] [blame] | 215 | return _compile(pattern, flags|T) |
| 216 | |
Guido van Rossum | 698280d | 2008-09-10 17:44:35 +0000 | [diff] [blame] | 217 | _alphanum_str = frozenset( |
| 218 | "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890") |
| 219 | _alphanum_bytes = frozenset( |
| 220 | b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890") |
Raymond Hettinger | ebb7f30 | 2005-09-12 22:50:37 +0000 | [diff] [blame] | 221 | |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 222 | def escape(pattern): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 223 | "Escape all non-alphanumeric characters in pattern." |
Guido van Rossum | 698280d | 2008-09-10 17:44:35 +0000 | [diff] [blame] | 224 | if isinstance(pattern, str): |
| 225 | alphanum = _alphanum_str |
| 226 | s = list(pattern) |
| 227 | for i in range(len(pattern)): |
| 228 | c = pattern[i] |
| 229 | if c not in alphanum: |
| 230 | if c == "\000": |
| 231 | s[i] = "\\000" |
| 232 | else: |
| 233 | s[i] = "\\" + c |
| 234 | return "".join(s) |
| 235 | else: |
| 236 | alphanum = _alphanum_bytes |
| 237 | s = [] |
| 238 | esc = ord(b"\\") |
| 239 | for c in pattern: |
| 240 | if c in alphanum: |
| 241 | s.append(c) |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 242 | else: |
Guido van Rossum | 698280d | 2008-09-10 17:44:35 +0000 | [diff] [blame] | 243 | if c == 0: |
| 244 | s.extend(b"\\000") |
| 245 | else: |
| 246 | s.append(esc) |
| 247 | s.append(c) |
| 248 | return bytes(s) |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 249 | |
| 250 | # -------------------------------------------------------------------- |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 251 | # internals |
Guido van Rossum | 7627c0d | 2000-03-31 14:58:54 +0000 | [diff] [blame] | 252 | |
Fredrik Lundh | 397a654 | 2001-10-18 19:30:16 +0000 | [diff] [blame] | 253 | _pattern_type = type(sre_compile.compile("", 0)) |
| 254 | |
Raymond Hettinger | d9e8cc6 | 2010-08-14 21:17:48 +0000 | [diff] [blame] | 255 | def _compile(pattern, flags): |
| 256 | return _compile_typed(type(pattern), pattern, flags) |
Gregory P. Smith | 5a63183 | 2010-07-27 05:31:29 +0000 | [diff] [blame] | 257 | |
Raymond Hettinger | d9e8cc6 | 2010-08-14 21:17:48 +0000 | [diff] [blame] | 258 | @functools.lru_cache(maxsize=500) |
| 259 | def _compile_typed(text_bytes_type, pattern, flags): |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 260 | # internal: compile pattern |
Just van Rossum | 12723ba | 2003-07-02 20:03:04 +0000 | [diff] [blame] | 261 | if isinstance(pattern, _pattern_type): |
Christian Heimes | 5fb7c2a | 2007-12-24 08:52:31 +0000 | [diff] [blame] | 262 | if flags: |
Guido van Rossum | 698280d | 2008-09-10 17:44:35 +0000 | [diff] [blame] | 263 | raise ValueError( |
| 264 | "Cannot process flags argument with a compiled pattern") |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 265 | return pattern |
Just van Rossum | 7490250 | 2003-07-02 21:37:16 +0000 | [diff] [blame] | 266 | if not sre_compile.isstring(pattern): |
Collin Winter | ce36ad8 | 2007-08-30 01:19:48 +0000 | [diff] [blame] | 267 | raise TypeError("first argument must be string or compiled pattern") |
Antoine Pitrou | 3060c45 | 2010-08-13 16:27:38 +0000 | [diff] [blame] | 268 | return sre_compile.compile(pattern, flags) |
Jeremy Hylton | b1aa195 | 2000-06-01 17:39:12 +0000 | [diff] [blame] | 269 | |
Raymond Hettinger | d9e8cc6 | 2010-08-14 21:17:48 +0000 | [diff] [blame] | 270 | @functools.lru_cache(maxsize=500) |
| 271 | def _compile_repl(repl, pattern): |
Fredrik Lundh | b25e1ad | 2001-03-22 15:50:10 +0000 | [diff] [blame] | 272 | # internal: compile replacement pattern |
Antoine Pitrou | 3060c45 | 2010-08-13 16:27:38 +0000 | [diff] [blame] | 273 | return sre_parse.parse_template(repl, pattern) |
Fredrik Lundh | b25e1ad | 2001-03-22 15:50:10 +0000 | [diff] [blame] | 274 | |
Fredrik Lundh | 5644b7f | 2000-09-21 17:03:25 +0000 | [diff] [blame] | 275 | def _expand(pattern, match, template): |
| 276 | # internal: match.expand implementation hook |
| 277 | template = sre_parse.parse_template(template, pattern) |
| 278 | return sre_parse.expand_template(template, match) |
| 279 | |
Fredrik Lundh | bec95b9 | 2001-10-21 16:47:57 +0000 | [diff] [blame] | 280 | def _subx(pattern, template): |
| 281 | # internal: pattern.sub/subn implementation helper |
Fredrik Lundh | dac5849 | 2001-10-21 21:48:30 +0000 | [diff] [blame] | 282 | template = _compile_repl(template, pattern) |
| 283 | if not template[0] and len(template[1]) == 1: |
Fredrik Lundh | bec95b9 | 2001-10-21 16:47:57 +0000 | [diff] [blame] | 284 | # literal replacement |
Fredrik Lundh | dac5849 | 2001-10-21 21:48:30 +0000 | [diff] [blame] | 285 | return template[1][0] |
| 286 | def filter(match, template=template): |
| 287 | return sre_parse.expand_template(template, match) |
| 288 | return filter |
Fredrik Lundh | 0640e11 | 2000-06-30 13:55:15 +0000 | [diff] [blame] | 289 | |
| 290 | # register myself for pickling |
| 291 | |
Alexandre Vassalotti | f7fa63d | 2008-05-11 08:55:36 +0000 | [diff] [blame] | 292 | import copyreg |
Fredrik Lundh | 0640e11 | 2000-06-30 13:55:15 +0000 | [diff] [blame] | 293 | |
| 294 | def _pickle(p): |
| 295 | return _compile, (p.pattern, p.flags) |
| 296 | |
Alexandre Vassalotti | f7fa63d | 2008-05-11 08:55:36 +0000 | [diff] [blame] | 297 | copyreg.pickle(_pattern_type, _pickle, _compile) |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 298 | |
| 299 | # -------------------------------------------------------------------- |
| 300 | # experimental stuff (see python-dev discussions for details) |
| 301 | |
| 302 | class Scanner: |
Fredrik Lundh | 1296a8d | 2001-10-21 18:04:11 +0000 | [diff] [blame] | 303 | def __init__(self, lexicon, flags=0): |
Fredrik Lundh | 29c4ba9 | 2000-08-01 18:20:07 +0000 | [diff] [blame] | 304 | from sre_constants import BRANCH, SUBPATTERN |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 305 | self.lexicon = lexicon |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 306 | # combine phrases into a compound pattern |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 307 | p = [] |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 308 | s = sre_parse.Pattern() |
Fredrik Lundh | 1296a8d | 2001-10-21 18:04:11 +0000 | [diff] [blame] | 309 | s.flags = flags |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 310 | for phrase, action in lexicon: |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 311 | p.append(sre_parse.SubPattern(s, [ |
Fredrik Lundh | 1296a8d | 2001-10-21 18:04:11 +0000 | [diff] [blame] | 312 | (SUBPATTERN, (len(p)+1, sre_parse.parse(phrase, flags))), |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 313 | ])) |
Christian Heimes | 2202f87 | 2008-02-06 14:31:34 +0000 | [diff] [blame] | 314 | s.groups = len(p)+1 |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 315 | p = sre_parse.SubPattern(s, [(BRANCH, (None, p))]) |
Fredrik Lundh | 8a3ebf8 | 2000-07-23 21:46:17 +0000 | [diff] [blame] | 316 | self.scanner = sre_compile.compile(p) |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 317 | def scan(self, string): |
| 318 | result = [] |
| 319 | append = result.append |
Fredrik Lundh | 1296a8d | 2001-10-21 18:04:11 +0000 | [diff] [blame] | 320 | match = self.scanner.scanner(string).match |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 321 | i = 0 |
| 322 | while 1: |
Fredrik Lundh | 1296a8d | 2001-10-21 18:04:11 +0000 | [diff] [blame] | 323 | m = match() |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 324 | if not m: |
| 325 | break |
| 326 | j = m.end() |
| 327 | if i == j: |
| 328 | break |
Fredrik Lundh | 1296a8d | 2001-10-21 18:04:11 +0000 | [diff] [blame] | 329 | action = self.lexicon[m.lastindex-1][1] |
Guido van Rossum | 698280d | 2008-09-10 17:44:35 +0000 | [diff] [blame] | 330 | if hasattr(action, "__call__"): |
Fredrik Lundh | 770617b | 2001-01-14 15:06:11 +0000 | [diff] [blame] | 331 | self.match = m |
Fredrik Lundh | 7cafe4d | 2000-07-02 17:33:27 +0000 | [diff] [blame] | 332 | action = action(self, m.group()) |
| 333 | if action is not None: |
| 334 | append(action) |
| 335 | i = j |
| 336 | return result, string[i:] |