Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 1 | .TH PCRE2TEST 1 "12 January 2022" "PCRE 10.40" |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 2 | .SH NAME |
| 3 | pcre2test - a program for testing Perl-compatible regular expressions. |
| 4 | .SH SYNOPSIS |
| 5 | .rs |
| 6 | .sp |
| 7 | .B pcre2test "[options] [input file [output file]]" |
| 8 | .sp |
| 9 | \fBpcre2test\fP is a test program for the PCRE2 regular expression libraries, |
| 10 | but it can also be used for experimenting with regular expressions. This |
| 11 | document describes the features of the test program; for details of the regular |
| 12 | expressions themselves, see the |
| 13 | .\" HREF |
| 14 | \fBpcre2pattern\fP |
| 15 | .\" |
| 16 | documentation. For details of the PCRE2 library function calls and their |
| 17 | options, see the |
| 18 | .\" HREF |
| 19 | \fBpcre2api\fP |
| 20 | .\" |
| 21 | documentation. |
| 22 | .P |
| 23 | The input for \fBpcre2test\fP is a sequence of regular expression patterns and |
| 24 | subject strings to be matched. There are also command lines for setting |
| 25 | defaults and controlling some special actions. The output shows the result of |
| 26 | each match attempt. Modifiers on external or internal command lines, the |
| 27 | patterns, and the subject lines specify PCRE2 function options, control how the |
| 28 | subject is processed, and what output is produced. |
| 29 | .P |
| 30 | There are many obscure modifiers, some of which are specifically designed for |
| 31 | use in conjunction with the test script and data files that are distributed as |
| 32 | part of PCRE2. All the modifiers are documented here, some without much |
| 33 | justification, but many of them are unlikely to be of use except when testing |
| 34 | the libraries. |
| 35 | . |
| 36 | . |
| 37 | .SH "PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES" |
| 38 | .rs |
| 39 | .sp |
| 40 | Different versions of the PCRE2 library can be built to support character |
| 41 | strings that are encoded in 8-bit, 16-bit, or 32-bit code units. One, two, or |
| 42 | all three of these libraries may be simultaneously installed. The |
| 43 | \fBpcre2test\fP program can be used to test all the libraries. However, its own |
| 44 | input and output are always in 8-bit format. When testing the 16-bit or 32-bit |
| 45 | libraries, patterns and subject strings are converted to 16-bit or 32-bit |
| 46 | format before being passed to the library functions. Results are converted back |
| 47 | to 8-bit code units for output. |
| 48 | .P |
| 49 | In the rest of this document, the names of library functions and structures |
Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 50 | are given in generic form, for example, \fBpcre2_compile()\fP. The actual |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 51 | names used in the libraries have a suffix _8, _16, or _32, as appropriate. |
| 52 | . |
| 53 | . |
| 54 | .\" HTML <a name="inputencoding"></a> |
| 55 | .SH "INPUT ENCODING" |
| 56 | .rs |
| 57 | .sp |
| 58 | Input to \fBpcre2test\fP is processed line by line, either by calling the C |
| 59 | library's \fBfgets()\fP function, or via the \fBlibreadline\fP or \fBlibedit\fP |
| 60 | library. In some Windows environments character 26 (hex 1A) causes an immediate |
| 61 | end of file, and no further data is read, so this character should be avoided |
| 62 | unless you really want that action. |
| 63 | .P |
| 64 | The input is processed using using C's string functions, so must not |
| 65 | contain binary zeros, even though in Unix-like environments, \fBfgets()\fP |
| 66 | treats any bytes other than newline as data characters. An error is generated |
| 67 | if a binary zero is encountered. By default subject lines are processed for |
| 68 | backslash escapes, which makes it possible to include any data value in strings |
| 69 | that are passed to the library for matching. For patterns, there is a facility |
| 70 | for specifying some or all of the 8-bit input characters as hexadecimal pairs, |
| 71 | which makes it possible to include binary zeros. |
| 72 | . |
| 73 | . |
| 74 | .SS "Input for the 16-bit and 32-bit libraries" |
| 75 | .rs |
| 76 | .sp |
| 77 | When testing the 16-bit or 32-bit libraries, there is a need to be able to |
| 78 | generate character code points greater than 255 in the strings that are passed |
| 79 | to the library. For subject lines, backslash escapes can be used. In addition, |
| 80 | when the \fButf\fP modifier (see |
| 81 | .\" HTML <a href="#optionmodifiers"> |
| 82 | .\" </a> |
| 83 | "Setting compilation options" |
| 84 | .\" |
| 85 | below) is set, the pattern and any following subject lines are interpreted as |
| 86 | UTF-8 strings and translated to UTF-16 or UTF-32 as appropriate. |
| 87 | .P |
| 88 | For non-UTF testing of wide characters, the \fButf8_input\fP modifier can be |
| 89 | used. This is mutually exclusive with \fButf\fP, and is allowed only in 16-bit |
| 90 | or 32-bit mode. It causes the pattern and following subject lines to be treated |
| 91 | as UTF-8 according to the original definition (RFC 2279), which allows for |
| 92 | character values up to 0x7fffffff. Each character is placed in one 16-bit or |
| 93 | 32-bit code unit (in the 16-bit case, values greater than 0xffff cause an error |
| 94 | to occur). |
| 95 | .P |
| 96 | UTF-8 (in its original definition) is not capable of encoding values greater |
| 97 | than 0x7fffffff, but such values can be handled by the 32-bit library. When |
| 98 | testing this library in non-UTF mode with \fButf8_input\fP set, if any |
| 99 | character is preceded by the byte 0xff (which is an invalid byte in UTF-8) |
| 100 | 0x80000000 is added to the character's value. This is the only way of passing |
| 101 | such code points in a pattern string. For subject strings, using an escape |
| 102 | sequence is preferable. |
| 103 | . |
| 104 | . |
| 105 | .SH "COMMAND LINE OPTIONS" |
| 106 | .rs |
| 107 | .TP 10 |
| 108 | \fB-8\fP |
| 109 | If the 8-bit library has been built, this option causes it to be used (this is |
| 110 | the default). If the 8-bit library has not been built, this option causes an |
| 111 | error. |
| 112 | .TP 10 |
| 113 | \fB-16\fP |
| 114 | If the 16-bit library has been built, this option causes it to be used. If only |
| 115 | the 16-bit library has been built, this is the default. If the 16-bit library |
| 116 | has not been built, this option causes an error. |
| 117 | .TP 10 |
| 118 | \fB-32\fP |
| 119 | If the 32-bit library has been built, this option causes it to be used. If only |
| 120 | the 32-bit library has been built, this is the default. If the 32-bit library |
| 121 | has not been built, this option causes an error. |
| 122 | .TP 10 |
| 123 | \fB-ac\fP |
| 124 | Behave as if each pattern has the \fBauto_callout\fP modifier, that is, insert |
| 125 | automatic callouts into every pattern that is compiled. |
| 126 | .TP 10 |
| 127 | \fB-AC\fP |
| 128 | As for \fB-ac\fP, but in addition behave as if each subject line has the |
| 129 | \fBcallout_extra\fP modifier, that is, show additional information from |
| 130 | callouts. |
| 131 | .TP 10 |
| 132 | \fB-b\fP |
| 133 | Behave as if each pattern has the \fBfullbincode\fP modifier; the full |
| 134 | internal binary form of the pattern is output after compilation. |
| 135 | .TP 10 |
| 136 | \fB-C\fP |
| 137 | Output the version number of the PCRE2 library, and all available information |
| 138 | about the optional features that are included, and then exit with zero exit |
| 139 | code. All other options are ignored. If both -C and -LM are present, whichever |
| 140 | is first is recognized. |
| 141 | .TP 10 |
| 142 | \fB-C\fP \fIoption\fP |
| 143 | Output information about a specific build-time option, then exit. This |
| 144 | functionality is intended for use in scripts such as \fBRunTest\fP. The |
| 145 | following options output the value and set the exit code as indicated: |
| 146 | .sp |
| 147 | ebcdic-nl the code for LF (= NL) in an EBCDIC environment: |
| 148 | 0x15 or 0x25 |
| 149 | 0 if used in an ASCII environment |
| 150 | exit code is always 0 |
| 151 | linksize the configured internal link size (2, 3, or 4) |
| 152 | exit code is set to the link size |
| 153 | newline the default newline setting: |
| 154 | CR, LF, CRLF, ANYCRLF, ANY, or NUL |
| 155 | exit code is always 0 |
| 156 | bsr the default setting for what \eR matches: |
| 157 | ANYCRLF or ANY |
| 158 | exit code is always 0 |
| 159 | .sp |
| 160 | The following options output 1 for true or 0 for false, and set the exit code |
| 161 | to the same value: |
| 162 | .sp |
| 163 | backslash-C \eC is supported (not locked out) |
| 164 | ebcdic compiled for an EBCDIC environment |
| 165 | jit just-in-time support is available |
| 166 | pcre2-16 the 16-bit library was built |
| 167 | pcre2-32 the 32-bit library was built |
| 168 | pcre2-8 the 8-bit library was built |
| 169 | unicode Unicode support is available |
| 170 | .sp |
| 171 | If an unknown option is given, an error message is output; the exit code is 0. |
| 172 | .TP 10 |
| 173 | \fB-d\fP |
| 174 | Behave as if each pattern has the \fBdebug\fP modifier; the internal |
| 175 | form and information about the compiled pattern is output after compilation; |
| 176 | \fB-d\fP is equivalent to \fB-b -i\fP. |
| 177 | .TP 10 |
| 178 | \fB-dfa\fP |
| 179 | Behave as if each subject line has the \fBdfa\fP modifier; matching is done |
| 180 | using the \fBpcre2_dfa_match()\fP function instead of the default |
| 181 | \fBpcre2_match()\fP. |
| 182 | .TP 10 |
| 183 | \fB-error\fP \fInumber[,number,...]\fP |
| 184 | Call \fBpcre2_get_error_message()\fP for each of the error numbers in the |
| 185 | comma-separated list, display the resulting messages on the standard output, |
| 186 | then exit with zero exit code. The numbers may be positive or negative. This is |
| 187 | a convenience facility for PCRE2 maintainers. |
| 188 | .TP 10 |
| 189 | \fB-help\fP |
| 190 | Output a brief summary these options and then exit. |
| 191 | .TP 10 |
| 192 | \fB-i\fP |
| 193 | Behave as if each pattern has the \fBinfo\fP modifier; information about the |
| 194 | compiled pattern is given after compilation. |
| 195 | .TP 10 |
| 196 | \fB-jit\fP |
| 197 | Behave as if each pattern line has the \fBjit\fP modifier; after successful |
| 198 | compilation, each pattern is passed to the just-in-time compiler, if available. |
| 199 | .TP 10 |
| 200 | \fB-jitfast\fP |
| 201 | Behave as if each pattern line has the \fBjitfast\fP modifier; after |
| 202 | successful compilation, each pattern is passed to the just-in-time compiler, if |
| 203 | available, and each subject line is passed directly to the JIT matcher via its |
| 204 | "fast path". |
| 205 | .TP 10 |
| 206 | \fB-jitverify\fP |
| 207 | Behave as if each pattern line has the \fBjitverify\fP modifier; after |
| 208 | successful compilation, each pattern is passed to the just-in-time compiler, if |
| 209 | available, and the use of JIT for matching is verified. |
| 210 | .TP 10 |
| 211 | \fB-LM\fP |
| 212 | List modifiers: write a list of available pattern and subject modifiers to the |
| 213 | standard output, then exit with zero exit code. All other options are ignored. |
Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 214 | If both -C and any -Lx options are present, whichever is first is recognized. |
| 215 | .TP 10 |
| 216 | \fB-LP\fP |
| 217 | List properties: write a list of recognized Unicode properties to the standard |
| 218 | output, then exit with zero exit code. All other options are ignored. If both |
| 219 | -C and any -Lx options are present, whichever is first is recognized. |
| 220 | .TP 10 |
| 221 | \fB-LS\fP |
| 222 | List scripts: write a list of recogized Unicode script names to the standard |
| 223 | output, then exit with zero exit code. All other options are ignored. If both |
| 224 | -C and any -Lx options are present, whichever is first is recognized. |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 225 | .TP 10 |
| 226 | \fB-pattern\fP \fImodifier-list\fP |
| 227 | Behave as if each pattern line contains the given modifiers. |
| 228 | .TP 10 |
| 229 | \fB-q\fP |
| 230 | Do not output the version number of \fBpcre2test\fP at the start of execution. |
| 231 | .TP 10 |
| 232 | \fB-S\fP \fIsize\fP |
| 233 | On Unix-like systems, set the size of the run-time stack to \fIsize\fP |
| 234 | mebibytes (units of 1024*1024 bytes). |
| 235 | .TP 10 |
| 236 | \fB-subject\fP \fImodifier-list\fP |
| 237 | Behave as if each subject line contains the given modifiers. |
| 238 | .TP 10 |
| 239 | \fB-t\fP |
| 240 | Run each compile and match many times with a timer, and output the resulting |
| 241 | times per compile or match. When JIT is used, separate times are given for the |
| 242 | initial compile and the JIT compile. You can control the number of iterations |
| 243 | that are used for timing by following \fB-t\fP with a number (as a separate |
| 244 | item on the command line). For example, "-t 1000" iterates 1000 times. The |
| 245 | default is to iterate 500,000 times. |
| 246 | .TP 10 |
| 247 | \fB-tm\fP |
| 248 | This is like \fB-t\fP except that it times only the matching phase, not the |
| 249 | compile phase. |
| 250 | .TP 10 |
| 251 | \fB-T\fP \fB-TM\fP |
| 252 | These behave like \fB-t\fP and \fB-tm\fP, but in addition, at the end of a run, |
| 253 | the total times for all compiles and matches are output. |
| 254 | .TP 10 |
| 255 | \fB-version\fP |
| 256 | Output the PCRE2 version number and then exit. |
| 257 | . |
| 258 | . |
| 259 | .SH "DESCRIPTION" |
| 260 | .rs |
| 261 | .sp |
| 262 | If \fBpcre2test\fP is given two filename arguments, it reads from the first and |
| 263 | writes to the second. If the first name is "-", input is taken from the |
| 264 | standard input. If \fBpcre2test\fP is given only one argument, it reads from |
| 265 | that file and writes to stdout. Otherwise, it reads from stdin and writes to |
| 266 | stdout. |
| 267 | .P |
| 268 | When \fBpcre2test\fP is built, a configuration option can specify that it |
| 269 | should be linked with the \fBlibreadline\fP or \fBlibedit\fP library. When this |
| 270 | is done, if the input is from a terminal, it is read using the \fBreadline()\fP |
| 271 | function. This provides line-editing and history facilities. The output from |
| 272 | the \fB-help\fP option states whether or not \fBreadline()\fP will be used. |
| 273 | .P |
| 274 | The program handles any number of tests, each of which consists of a set of |
| 275 | input lines. Each set starts with a regular expression pattern, followed by any |
| 276 | number of subject lines to be matched against that pattern. In between sets of |
| 277 | test data, command lines that begin with # may appear. This file format, with |
| 278 | some restrictions, can also be processed by the \fBperltest.sh\fP script that |
| 279 | is distributed with PCRE2 as a means of checking that the behaviour of PCRE2 |
| 280 | and Perl is the same. For a specification of \fBperltest.sh\fP, see the |
| 281 | comments near its beginning. See also the #perltest command below. |
| 282 | .P |
| 283 | When the input is a terminal, \fBpcre2test\fP prompts for each line of input, |
| 284 | using "re>" to prompt for regular expression patterns, and "data>" to prompt |
| 285 | for subject lines. Command lines starting with # can be entered only in |
| 286 | response to the "re>" prompt. |
| 287 | .P |
| 288 | Each subject line is matched separately and independently. If you want to do |
| 289 | multi-line matches, you have to use the \en escape sequence (or \er or \er\en, |
| 290 | etc., depending on the newline setting) in a single line of input to encode the |
| 291 | newline sequences. There is no limit on the length of subject lines; the input |
| 292 | buffer is automatically extended if it is too small. There are replication |
| 293 | features that makes it possible to generate long repetitive pattern or subject |
| 294 | lines without having to supply them explicitly. |
| 295 | .P |
| 296 | An empty line or the end of the file signals the end of the subject lines for a |
| 297 | test, at which point a new pattern or command line is expected if there is |
| 298 | still input to be read. |
| 299 | . |
| 300 | . |
| 301 | .SH "COMMAND LINES" |
| 302 | .rs |
| 303 | .sp |
| 304 | In between sets of test data, a line that begins with # is interpreted as a |
| 305 | command line. If the first character is followed by white space or an |
| 306 | exclamation mark, the line is treated as a comment, and ignored. Otherwise, the |
| 307 | following commands are recognized: |
| 308 | .sp |
| 309 | #forbid_utf |
| 310 | .sp |
| 311 | Subsequent patterns automatically have the PCRE2_NEVER_UTF and PCRE2_NEVER_UCP |
| 312 | options set, which locks out the use of the PCRE2_UTF and PCRE2_UCP options and |
| 313 | the use of (*UTF) and (*UCP) at the start of patterns. This command also forces |
| 314 | an error if a subsequent pattern contains any occurrences of \eP, \ep, or \eX, |
| 315 | which are still supported when PCRE2_UTF is not set, but which require Unicode |
| 316 | property support to be included in the library. |
| 317 | .P |
| 318 | This is a trigger guard that is used in test files to ensure that UTF or |
| 319 | Unicode property tests are not accidentally added to files that are used when |
| 320 | Unicode support is not included in the library. Setting PCRE2_NEVER_UTF and |
| 321 | PCRE2_NEVER_UCP as a default can also be obtained by the use of \fB#pattern\fP; |
| 322 | the difference is that \fB#forbid_utf\fP cannot be unset, and the automatic |
| 323 | options are not displayed in pattern information, to avoid cluttering up test |
| 324 | output. |
| 325 | .sp |
| 326 | #load <filename> |
| 327 | .sp |
| 328 | This command is used to load a set of precompiled patterns from a file, as |
| 329 | described in the section entitled "Saving and restoring compiled patterns" |
| 330 | .\" HTML <a href="#saverestore"> |
| 331 | .\" </a> |
| 332 | below. |
| 333 | .\" |
| 334 | .sp |
| 335 | #loadtables <filename> |
| 336 | .sp |
| 337 | This command is used to load a set of binary character tables that can be |
| 338 | accessed by the tables=3 qualifier. Such tables can be created by the |
| 339 | \fBpcre2_dftables\fP program with the -b option. |
| 340 | .sp |
| 341 | #newline_default [<newline-list>] |
| 342 | .sp |
| 343 | When PCRE2 is built, a default newline convention can be specified. This |
| 344 | determines which characters and/or character pairs are recognized as indicating |
| 345 | a newline in a pattern or subject string. The default can be overridden when a |
| 346 | pattern is compiled. The standard test files contain tests of various newline |
| 347 | conventions, but the majority of the tests expect a single linefeed to be |
| 348 | recognized as a newline by default. Without special action the tests would fail |
| 349 | when PCRE2 is compiled with either CR or CRLF as the default newline. |
| 350 | .P |
| 351 | The #newline_default command specifies a list of newline types that are |
| 352 | acceptable as the default. The types must be one of CR, LF, CRLF, ANYCRLF, |
| 353 | ANY, or NUL (in upper or lower case), for example: |
| 354 | .sp |
| 355 | #newline_default LF Any anyCRLF |
| 356 | .sp |
| 357 | If the default newline is in the list, this command has no effect. Otherwise, |
| 358 | except when testing the POSIX API, a \fBnewline\fP modifier that specifies the |
| 359 | first newline convention in the list (LF in the above example) is added to any |
| 360 | pattern that does not already have a \fBnewline\fP modifier. If the newline |
| 361 | list is empty, the feature is turned off. This command is present in a number |
| 362 | of the standard test input files. |
| 363 | .P |
| 364 | When the POSIX API is being tested there is no way to override the default |
| 365 | newline convention, though it is possible to set the newline convention from |
| 366 | within the pattern. A warning is given if the \fBposix\fP or \fBposix_nosub\fP |
| 367 | modifier is used when \fB#newline_default\fP would set a default for the |
| 368 | non-POSIX API. |
| 369 | .sp |
| 370 | #pattern <modifier-list> |
| 371 | .sp |
| 372 | This command sets a default modifier list that applies to all subsequent |
| 373 | patterns. Modifiers on a pattern can change these settings. |
| 374 | .sp |
| 375 | #perltest |
| 376 | .sp |
| 377 | This line is used in test files that can also be processed by \fBperltest.sh\fP |
| 378 | to confirm that Perl gives the same results as PCRE2. Subsequent tests are |
| 379 | checked for the use of \fBpcre2test\fP features that are incompatible with the |
| 380 | \fBperltest.sh\fP script. |
| 381 | .P |
| 382 | Patterns must use '/' as their delimiter, and only certain modifiers are |
| 383 | supported. Comment lines, #pattern commands, and #subject commands that set or |
| 384 | unset "mark" are recognized and acted on. The #perltest, #forbid_utf, and |
| 385 | #newline_default commands, which are needed in the relevant pcre2test files, |
| 386 | are silently ignored. All other command lines are ignored, but give a warning |
| 387 | message. The \fB#perltest\fP command helps detect tests that are accidentally |
| 388 | put in the wrong file or use the wrong delimiter. For more details of the |
| 389 | \fBperltest.sh\fP script see the comments it contains. |
| 390 | .sp |
| 391 | #pop [<modifiers>] |
| 392 | #popcopy [<modifiers>] |
| 393 | .sp |
| 394 | These commands are used to manipulate the stack of compiled patterns, as |
| 395 | described in the section entitled "Saving and restoring compiled patterns" |
| 396 | .\" HTML <a href="#saverestore"> |
| 397 | .\" </a> |
| 398 | below. |
| 399 | .\" |
| 400 | .sp |
| 401 | #save <filename> |
| 402 | .sp |
| 403 | This command is used to save a set of compiled patterns to a file, as described |
| 404 | in the section entitled "Saving and restoring compiled patterns" |
| 405 | .\" HTML <a href="#saverestore"> |
| 406 | .\" </a> |
| 407 | below. |
| 408 | .\" |
| 409 | .sp |
| 410 | #subject <modifier-list> |
| 411 | .sp |
| 412 | This command sets a default modifier list that applies to all subsequent |
| 413 | subject lines. Modifiers on a subject line can change these settings. |
| 414 | . |
| 415 | . |
| 416 | .SH "MODIFIER SYNTAX" |
| 417 | .rs |
| 418 | .sp |
| 419 | Modifier lists are used with both pattern and subject lines. Items in a list |
| 420 | are separated by commas followed by optional white space. Trailing whitespace |
| 421 | in a modifier list is ignored. Some modifiers may be given for both patterns |
| 422 | and subject lines, whereas others are valid only for one or the other. Each |
| 423 | modifier has a long name, for example "anchored", and some of them must be |
| 424 | followed by an equals sign and a value, for example, "offset=12". Values cannot |
| 425 | contain comma characters, but may contain spaces. Modifiers that do not take |
| 426 | values may be preceded by a minus sign to turn off a previous setting. |
| 427 | .P |
| 428 | A few of the more common modifiers can also be specified as single letters, for |
| 429 | example "i" for "caseless". In documentation, following the Perl convention, |
| 430 | these are written with a slash ("the /i modifier") for clarity. Abbreviated |
| 431 | modifiers must all be concatenated in the first item of a modifier list. If the |
| 432 | first item is not recognized as a long modifier name, it is interpreted as a |
| 433 | sequence of these abbreviations. For example: |
| 434 | .sp |
| 435 | /abc/ig,newline=cr,jit=3 |
| 436 | .sp |
| 437 | This is a pattern line whose modifier list starts with two one-letter modifiers |
| 438 | (/i and /g). The lower-case abbreviated modifiers are the same as used in Perl. |
| 439 | . |
| 440 | . |
| 441 | .SH "PATTERN SYNTAX" |
| 442 | .rs |
| 443 | .sp |
| 444 | A pattern line must start with one of the following characters (common symbols, |
| 445 | excluding pattern meta-characters): |
| 446 | .sp |
| 447 | / ! " ' ` - = _ : ; , % & @ ~ |
| 448 | .sp |
| 449 | This is interpreted as the pattern's delimiter. A regular expression may be |
| 450 | continued over several input lines, in which case the newline characters are |
| 451 | included within it. It is possible to include the delimiter as a literal within |
| 452 | the pattern by escaping it with a backslash, for example |
| 453 | .sp |
| 454 | /abc\e/def/ |
| 455 | .sp |
| 456 | If you do this, the escape and the delimiter form part of the pattern, but |
| 457 | since the delimiters are all non-alphanumeric, the inclusion of the backslash |
| 458 | does not affect the pattern's interpretation. Note, however, that this trick |
| 459 | does not work within \eQ...\eE literal bracketing because the backslash will |
| 460 | itself be interpreted as a literal. If the terminating delimiter is immediately |
| 461 | followed by a backslash, for example, |
| 462 | .sp |
| 463 | /abc/\e |
| 464 | .sp |
| 465 | then a backslash is added to the end of the pattern. This is done to provide a |
| 466 | way of testing the error condition that arises if a pattern finishes with a |
| 467 | backslash, because |
| 468 | .sp |
| 469 | /abc\e/ |
| 470 | .sp |
| 471 | is interpreted as the first line of a pattern that starts with "abc/", causing |
| 472 | pcre2test to read the next line as a continuation of the regular expression. |
| 473 | .P |
| 474 | A pattern can be followed by a modifier list (details below). |
| 475 | . |
| 476 | . |
| 477 | .SH "SUBJECT LINE SYNTAX" |
| 478 | .rs |
| 479 | .sp |
| 480 | Before each subject line is passed to \fBpcre2_match()\fP, |
| 481 | \fBpcre2_dfa_match()\fP, or \fBpcre2_jit_match()\fP, leading and trailing white |
| 482 | space is removed, and the line is scanned for backslash escapes, unless the |
| 483 | \fBsubject_literal\fP modifier was set for the pattern. The following provide a |
| 484 | means of encoding non-printing characters in a visible way: |
| 485 | .sp |
| 486 | \ea alarm (BEL, \ex07) |
| 487 | \eb backspace (\ex08) |
| 488 | \ee escape (\ex27) |
| 489 | \ef form feed (\ex0c) |
| 490 | \en newline (\ex0a) |
| 491 | \er carriage return (\ex0d) |
| 492 | \et tab (\ex09) |
| 493 | \ev vertical tab (\ex0b) |
| 494 | \ennn octal character (up to 3 octal digits); always |
| 495 | a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode |
| 496 | \eo{dd...} octal character (any number of octal digits} |
| 497 | \exhh hexadecimal byte (up to 2 hex digits) |
| 498 | \ex{hh...} hexadecimal character (any number of hex digits) |
| 499 | .sp |
| 500 | The use of \ex{hh...} is not dependent on the use of the \fButf\fP modifier on |
| 501 | the pattern. It is recognized always. There may be any number of hexadecimal |
| 502 | digits inside the braces; invalid values provoke error messages. |
| 503 | .P |
| 504 | Note that \exhh specifies one byte rather than one character in UTF-8 mode; |
| 505 | this makes it possible to construct invalid UTF-8 sequences for testing |
| 506 | purposes. On the other hand, \ex{hh} is interpreted as a UTF-8 character in |
| 507 | UTF-8 mode, generating more than one byte if the value is greater than 127. |
| 508 | When testing the 8-bit library not in UTF-8 mode, \ex{hh} generates one byte |
| 509 | for values less than 256, and causes an error for greater values. |
| 510 | .P |
| 511 | In UTF-16 mode, all 4-digit \ex{hhhh} values are accepted. This makes it |
| 512 | possible to construct invalid UTF-16 sequences for testing purposes. |
| 513 | .P |
| 514 | In UTF-32 mode, all 4- to 8-digit \ex{...} values are accepted. This makes it |
| 515 | possible to construct invalid UTF-32 sequences for testing purposes. |
| 516 | .P |
| 517 | There is a special backslash sequence that specifies replication of one or more |
| 518 | characters: |
| 519 | .sp |
| 520 | \e[<characters>]{<count>} |
| 521 | .sp |
| 522 | This makes it possible to test long strings without having to provide them as |
| 523 | part of the file. For example: |
| 524 | .sp |
| 525 | \e[abc]{4} |
| 526 | .sp |
| 527 | is converted to "abcabcabcabc". This feature does not support nesting. To |
| 528 | include a closing square bracket in the characters, code it as \ex5D. |
| 529 | .P |
| 530 | A backslash followed by an equals sign marks the end of the subject string and |
| 531 | the start of a modifier list. For example: |
| 532 | .sp |
| 533 | abc\e=notbol,notempty |
| 534 | .sp |
| 535 | If the subject string is empty and \e= is followed by whitespace, the line is |
| 536 | treated as a comment line, and is not used for matching. For example: |
| 537 | .sp |
| 538 | \e= This is a comment. |
| 539 | abc\e= This is an invalid modifier list. |
| 540 | .sp |
| 541 | A backslash followed by any other non-alphanumeric character just escapes that |
| 542 | character. A backslash followed by anything else causes an error. However, if |
| 543 | the very last character in the line is a backslash (and there is no modifier |
| 544 | list), it is ignored. This gives a way of passing an empty line as data, since |
| 545 | a real empty line terminates the data input. |
| 546 | .P |
| 547 | If the \fBsubject_literal\fP modifier is set for a pattern, all subject lines |
| 548 | that follow are treated as literals, with no special treatment of backslashes. |
| 549 | No replication is possible, and any subject modifiers must be set as defaults |
| 550 | by a \fB#subject\fP command. |
| 551 | . |
| 552 | . |
| 553 | .SH "PATTERN MODIFIERS" |
| 554 | .rs |
| 555 | .sp |
| 556 | There are several types of modifier that can appear in pattern lines. Except |
| 557 | where noted below, they may also be used in \fB#pattern\fP commands. A |
| 558 | pattern's modifier list can add to or override default modifiers that were set |
| 559 | by a previous \fB#pattern\fP command. |
| 560 | . |
| 561 | . |
| 562 | .\" HTML <a name="optionmodifiers"></a> |
| 563 | .SS "Setting compilation options" |
| 564 | .rs |
| 565 | .sp |
| 566 | The following modifiers set options for \fBpcre2_compile()\fP. Most of them set |
| 567 | bits in the options argument of that function, but those whose names start with |
| 568 | PCRE2_EXTRA are additional options that are set in the compile context. For the |
| 569 | main options, there are some single-letter abbreviations that are the same as |
| 570 | Perl options. There is special handling for /x: if a second x is present, |
| 571 | PCRE2_EXTENDED is converted into PCRE2_EXTENDED_MORE as in Perl. A third |
| 572 | appearance adds PCRE2_EXTENDED as well, though this makes no difference to the |
| 573 | way \fBpcre2_compile()\fP behaves. See |
| 574 | .\" HREF |
| 575 | \fBpcre2api\fP |
| 576 | .\" |
| 577 | for a description of the effects of these options. |
| 578 | .sp |
| 579 | allow_empty_class set PCRE2_ALLOW_EMPTY_CLASS |
| 580 | allow_lookaround_bsk set PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK |
| 581 | allow_surrogate_escapes set PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES |
| 582 | alt_bsux set PCRE2_ALT_BSUX |
| 583 | alt_circumflex set PCRE2_ALT_CIRCUMFLEX |
| 584 | alt_verbnames set PCRE2_ALT_VERBNAMES |
| 585 | anchored set PCRE2_ANCHORED |
| 586 | auto_callout set PCRE2_AUTO_CALLOUT |
| 587 | bad_escape_is_literal set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL |
| 588 | /i caseless set PCRE2_CASELESS |
| 589 | dollar_endonly set PCRE2_DOLLAR_ENDONLY |
| 590 | /s dotall set PCRE2_DOTALL |
| 591 | dupnames set PCRE2_DUPNAMES |
| 592 | endanchored set PCRE2_ENDANCHORED |
| 593 | escaped_cr_is_lf set PCRE2_EXTRA_ESCAPED_CR_IS_LF |
| 594 | /x extended set PCRE2_EXTENDED |
| 595 | /xx extended_more set PCRE2_EXTENDED_MORE |
| 596 | extra_alt_bsux set PCRE2_EXTRA_ALT_BSUX |
| 597 | firstline set PCRE2_FIRSTLINE |
| 598 | literal set PCRE2_LITERAL |
| 599 | match_line set PCRE2_EXTRA_MATCH_LINE |
| 600 | match_invalid_utf set PCRE2_MATCH_INVALID_UTF |
| 601 | match_unset_backref set PCRE2_MATCH_UNSET_BACKREF |
| 602 | match_word set PCRE2_EXTRA_MATCH_WORD |
| 603 | /m multiline set PCRE2_MULTILINE |
| 604 | never_backslash_c set PCRE2_NEVER_BACKSLASH_C |
| 605 | never_ucp set PCRE2_NEVER_UCP |
| 606 | never_utf set PCRE2_NEVER_UTF |
| 607 | /n no_auto_capture set PCRE2_NO_AUTO_CAPTURE |
| 608 | no_auto_possess set PCRE2_NO_AUTO_POSSESS |
| 609 | no_dotstar_anchor set PCRE2_NO_DOTSTAR_ANCHOR |
| 610 | no_start_optimize set PCRE2_NO_START_OPTIMIZE |
| 611 | no_utf_check set PCRE2_NO_UTF_CHECK |
| 612 | ucp set PCRE2_UCP |
| 613 | ungreedy set PCRE2_UNGREEDY |
| 614 | use_offset_limit set PCRE2_USE_OFFSET_LIMIT |
| 615 | utf set PCRE2_UTF |
| 616 | .sp |
| 617 | As well as turning on the PCRE2_UTF option, the \fButf\fP modifier causes all |
| 618 | non-printing characters in output strings to be printed using the \ex{hh...} |
| 619 | notation. Otherwise, those less than 0x100 are output in hex without the curly |
| 620 | brackets. Setting \fButf\fP in 16-bit or 32-bit mode also causes pattern and |
| 621 | subject strings to be translated to UTF-16 or UTF-32, respectively, before |
| 622 | being passed to library functions. |
| 623 | . |
| 624 | . |
| 625 | .\" HTML <a name="controlmodifiers"></a> |
| 626 | .SS "Setting compilation controls" |
| 627 | .rs |
| 628 | .sp |
| 629 | The following modifiers affect the compilation process or request information |
| 630 | about the pattern. There are single-letter abbreviations for some that are |
| 631 | heavily used in the test files. |
| 632 | .sp |
| 633 | bsr=[anycrlf|unicode] specify \eR handling |
| 634 | /B bincode show binary code without lengths |
| 635 | callout_info show callout information |
| 636 | convert=<options> request foreign pattern conversion |
| 637 | convert_glob_escape=c set glob escape character |
| 638 | convert_glob_separator=c set glob separator character |
| 639 | convert_length set convert buffer length |
| 640 | debug same as info,fullbincode |
| 641 | framesize show matching frame size |
| 642 | fullbincode show binary code with lengths |
| 643 | /I info show info about compiled pattern |
| 644 | hex unquoted characters are hexadecimal |
| 645 | jit[=<number>] use JIT |
| 646 | jitfast use JIT fast path |
| 647 | jitverify verify JIT use |
| 648 | locale=<name> use this locale |
| 649 | max_pattern_length=<n> set the maximum pattern length |
| 650 | memory show memory used |
| 651 | newline=<type> set newline type |
| 652 | null_context compile with a NULL context |
| 653 | parens_nest_limit=<n> set maximum parentheses depth |
| 654 | posix use the POSIX API |
| 655 | posix_nosub use the POSIX API with REG_NOSUB |
| 656 | push push compiled pattern onto the stack |
| 657 | pushcopy push a copy onto the stack |
| 658 | stackguard=<number> test the stackguard feature |
| 659 | subject_literal treat all subject lines as literal |
| 660 | tables=[0|1|2|3] select internal tables |
| 661 | use_length do not zero-terminate the pattern |
| 662 | utf8_input treat input as UTF-8 |
| 663 | .sp |
| 664 | The effects of these modifiers are described in the following sections. |
| 665 | . |
| 666 | . |
| 667 | .SS "Newline and \eR handling" |
| 668 | .rs |
| 669 | .sp |
| 670 | The \fBbsr\fP modifier specifies what \eR in a pattern should match. If it is |
| 671 | set to "anycrlf", \eR matches CR, LF, or CRLF only. If it is set to "unicode", |
| 672 | \eR matches any Unicode newline sequence. The default can be specified when |
| 673 | PCRE2 is built; if it is not, the default is set to Unicode. |
| 674 | .P |
| 675 | The \fBnewline\fP modifier specifies which characters are to be interpreted as |
| 676 | newlines, both in the pattern and in subject lines. The type must be one of CR, |
| 677 | LF, CRLF, ANYCRLF, ANY, or NUL (in upper or lower case). |
| 678 | . |
| 679 | . |
| 680 | .SS "Information about a pattern" |
| 681 | .rs |
| 682 | .sp |
| 683 | The \fBdebug\fP modifier is a shorthand for \fBinfo,fullbincode\fP, requesting |
| 684 | all available information. |
| 685 | .P |
| 686 | The \fBbincode\fP modifier causes a representation of the compiled code to be |
| 687 | output after compilation. This information does not contain length and offset |
| 688 | values, which ensures that the same output is generated for different internal |
| 689 | link sizes and different code unit widths. By using \fBbincode\fP, the same |
| 690 | regression tests can be used in different environments. |
| 691 | .P |
| 692 | The \fBfullbincode\fP modifier, by contrast, \fIdoes\fP include length and |
| 693 | offset values. This is used in a few special tests that run only for specific |
| 694 | code unit widths and link sizes, and is also useful for one-off tests. |
| 695 | .P |
| 696 | The \fBinfo\fP modifier requests information about the compiled pattern |
| 697 | (whether it is anchored, has a fixed first character, and so on). The |
| 698 | information is obtained from the \fBpcre2_pattern_info()\fP function. Here are |
| 699 | some typical examples: |
| 700 | .sp |
| 701 | re> /(?i)(^a|^b)/m,info |
| 702 | Capture group count = 1 |
| 703 | Compile options: multiline |
| 704 | Overall options: caseless multiline |
| 705 | First code unit at start or follows newline |
| 706 | Subject length lower bound = 1 |
| 707 | .sp |
| 708 | re> /(?i)abc/info |
| 709 | Capture group count = 0 |
| 710 | Compile options: <none> |
| 711 | Overall options: caseless |
| 712 | First code unit = 'a' (caseless) |
| 713 | Last code unit = 'c' (caseless) |
| 714 | Subject length lower bound = 3 |
| 715 | .sp |
| 716 | "Compile options" are those specified by modifiers; "overall options" have |
| 717 | added options that are taken or deduced from the pattern. If both sets of |
| 718 | options are the same, just a single "options" line is output; if there are no |
| 719 | options, the line is omitted. "First code unit" is where any match must start; |
| 720 | if there is more than one they are listed as "starting code units". "Last code |
| 721 | unit" is the last literal code unit that must be present in any match. This is |
| 722 | not necessarily the last character. These lines are omitted if no starting or |
| 723 | ending code units are recorded. The subject length line is omitted when |
| 724 | \fBno_start_optimize\fP is set because the minimum length is not calculated |
| 725 | when it can never be used. |
| 726 | .P |
| 727 | The \fBframesize\fP modifier shows the size, in bytes, of the storage frames |
| 728 | used by \fBpcre2_match()\fP for handling backtracking. The size depends on the |
| 729 | number of capturing parentheses in the pattern. |
| 730 | .P |
| 731 | The \fBcallout_info\fP modifier requests information about all the callouts in |
| 732 | the pattern. A list of them is output at the end of any other information that |
| 733 | is requested. For each callout, either its number or string is given, followed |
| 734 | by the item that follows it in the pattern. |
| 735 | . |
| 736 | . |
| 737 | .SS "Passing a NULL context" |
| 738 | .rs |
| 739 | .sp |
| 740 | Normally, \fBpcre2test\fP passes a context block to \fBpcre2_compile()\fP. If |
| 741 | the \fBnull_context\fP modifier is set, however, NULL is passed. This is for |
| 742 | testing that \fBpcre2_compile()\fP behaves correctly in this case (it uses |
| 743 | default values). |
| 744 | . |
| 745 | . |
| 746 | .SS "Specifying pattern characters in hexadecimal" |
| 747 | .rs |
| 748 | .sp |
| 749 | The \fBhex\fP modifier specifies that the characters of the pattern, except for |
| 750 | substrings enclosed in single or double quotes, are to be interpreted as pairs |
| 751 | of hexadecimal digits. This feature is provided as a way of creating patterns |
| 752 | that contain binary zeros and other non-printing characters. White space is |
| 753 | permitted between pairs of digits. For example, this pattern contains three |
| 754 | characters: |
| 755 | .sp |
| 756 | /ab 32 59/hex |
| 757 | .sp |
| 758 | Parts of such a pattern are taken literally if quoted. This pattern contains |
| 759 | nine characters, only two of which are specified in hexadecimal: |
| 760 | .sp |
| 761 | /ab "literal" 32/hex |
| 762 | .sp |
| 763 | Either single or double quotes may be used. There is no way of including |
| 764 | the delimiter within a substring. The \fBhex\fP and \fBexpand\fP modifiers are |
| 765 | mutually exclusive. |
| 766 | . |
| 767 | . |
| 768 | .SS "Specifying the pattern's length" |
| 769 | .rs |
| 770 | .sp |
| 771 | By default, patterns are passed to the compiling functions as zero-terminated |
| 772 | strings but can be passed by length instead of being zero-terminated. The |
| 773 | \fBuse_length\fP modifier causes this to happen. Using a length happens |
| 774 | automatically (whether or not \fBuse_length\fP is set) when \fBhex\fP is set, |
| 775 | because patterns specified in hexadecimal may contain binary zeros. |
| 776 | .P |
| 777 | If \fBhex\fP or \fBuse_length\fP is used with the POSIX wrapper API (see |
| 778 | .\" HTML <a href="#posixwrapper"> |
| 779 | .\" </a> |
| 780 | "Using the POSIX wrapper API" |
| 781 | .\" |
| 782 | below), the REG_PEND extension is used to pass the pattern's length. |
| 783 | . |
| 784 | . |
| 785 | .SS "Specifying wide characters in 16-bit and 32-bit modes" |
| 786 | .rs |
| 787 | .sp |
| 788 | In 16-bit and 32-bit modes, all input is automatically treated as UTF-8 and |
| 789 | translated to UTF-16 or UTF-32 when the \fButf\fP modifier is set. For testing |
| 790 | the 16-bit and 32-bit libraries in non-UTF mode, the \fButf8_input\fP modifier |
| 791 | can be used. It is mutually exclusive with \fButf\fP. Input lines are |
| 792 | interpreted as UTF-8 as a means of specifying wide characters. More details are |
| 793 | given in |
| 794 | .\" HTML <a href="#inputencoding"> |
| 795 | .\" </a> |
| 796 | "Input encoding" |
| 797 | .\" |
| 798 | above. |
| 799 | . |
| 800 | . |
| 801 | .SS "Generating long repetitive patterns" |
| 802 | .rs |
| 803 | .sp |
| 804 | Some tests use long patterns that are very repetitive. Instead of creating a |
| 805 | very long input line for such a pattern, you can use a special repetition |
| 806 | feature, similar to the one described for subject lines above. If the |
| 807 | \fBexpand\fP modifier is present on a pattern, parts of the pattern that have |
| 808 | the form |
| 809 | .sp |
| 810 | \e[<characters>]{<count>} |
| 811 | .sp |
| 812 | are expanded before the pattern is passed to \fBpcre2_compile()\fP. For |
| 813 | example, \e[AB]{6000} is expanded to "ABAB..." 6000 times. This construction |
| 814 | cannot be nested. An initial "\e[" sequence is recognized only if "]{" followed |
| 815 | by decimal digits and "}" is found later in the pattern. If not, the characters |
| 816 | remain in the pattern unaltered. The \fBexpand\fP and \fBhex\fP modifiers are |
| 817 | mutually exclusive. |
| 818 | .P |
| 819 | If part of an expanded pattern looks like an expansion, but is really part of |
| 820 | the actual pattern, unwanted expansion can be avoided by giving two values in |
| 821 | the quantifier. For example, \e[AB]{6000,6000} is not recognized as an |
| 822 | expansion item. |
| 823 | .P |
| 824 | If the \fBinfo\fP modifier is set on an expanded pattern, the result of the |
| 825 | expansion is included in the information that is output. |
| 826 | . |
| 827 | . |
| 828 | .SS "JIT compilation" |
| 829 | .rs |
| 830 | .sp |
| 831 | Just-in-time (JIT) compiling is a heavyweight optimization that can greatly |
| 832 | speed up pattern matching. See the |
| 833 | .\" HREF |
| 834 | \fBpcre2jit\fP |
| 835 | .\" |
| 836 | documentation for details. JIT compiling happens, optionally, after a pattern |
| 837 | has been successfully compiled into an internal form. The JIT compiler converts |
| 838 | this to optimized machine code. It needs to know whether the match-time options |
| 839 | PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT are going to be used, because |
| 840 | different code is generated for the different cases. See the \fBpartial\fP |
| 841 | modifier in "Subject Modifiers" |
| 842 | .\" HTML <a href="#subjectmodifiers"> |
| 843 | .\" </a> |
| 844 | below |
| 845 | .\" |
| 846 | for details of how these options are specified for each match attempt. |
| 847 | .P |
| 848 | JIT compilation is requested by the \fBjit\fP pattern modifier, which may |
| 849 | optionally be followed by an equals sign and a number in the range 0 to 7. |
| 850 | The three bits that make up the number specify which of the three JIT operating |
| 851 | modes are to be compiled: |
| 852 | .sp |
| 853 | 1 compile JIT code for non-partial matching |
| 854 | 2 compile JIT code for soft partial matching |
| 855 | 4 compile JIT code for hard partial matching |
| 856 | .sp |
| 857 | The possible values for the \fBjit\fP modifier are therefore: |
| 858 | .sp |
| 859 | 0 disable JIT |
| 860 | 1 normal matching only |
| 861 | 2 soft partial matching only |
| 862 | 3 normal and soft partial matching |
| 863 | 4 hard partial matching only |
| 864 | 6 soft and hard partial matching only |
| 865 | 7 all three modes |
| 866 | .sp |
| 867 | If no number is given, 7 is assumed. The phrase "partial matching" means a call |
| 868 | to \fBpcre2_match()\fP with either the PCRE2_PARTIAL_SOFT or the |
| 869 | PCRE2_PARTIAL_HARD option set. Note that such a call may return a complete |
| 870 | match; the options enable the possibility of a partial match, but do not |
| 871 | require it. Note also that if you request JIT compilation only for partial |
| 872 | matching (for example, jit=2) but do not set the \fBpartial\fP modifier on a |
| 873 | subject line, that match will not use JIT code because none was compiled for |
| 874 | non-partial matching. |
| 875 | .P |
| 876 | If JIT compilation is successful, the compiled JIT code will automatically be |
| 877 | used when an appropriate type of match is run, except when incompatible |
| 878 | run-time options are specified. For more details, see the |
| 879 | .\" HREF |
| 880 | \fBpcre2jit\fP |
| 881 | .\" |
| 882 | documentation. See also the \fBjitstack\fP modifier below for a way of |
| 883 | setting the size of the JIT stack. |
| 884 | .P |
| 885 | If the \fBjitfast\fP modifier is specified, matching is done using the JIT |
| 886 | "fast path" interface, \fBpcre2_jit_match()\fP, which skips some of the sanity |
| 887 | checks that are done by \fBpcre2_match()\fP, and of course does not work when |
| 888 | JIT is not supported. If \fBjitfast\fP is specified without \fBjit\fP, jit=7 is |
| 889 | assumed. |
| 890 | .P |
| 891 | If the \fBjitverify\fP modifier is specified, information about the compiled |
| 892 | pattern shows whether JIT compilation was or was not successful. If |
| 893 | \fBjitverify\fP is specified without \fBjit\fP, jit=7 is assumed. If JIT |
| 894 | compilation is successful when \fBjitverify\fP is set, the text "(JIT)" is |
| 895 | added to the first output line after a match or non match when JIT-compiled |
| 896 | code was actually used in the match. |
| 897 | . |
| 898 | . |
| 899 | .SS "Setting a locale" |
| 900 | .rs |
| 901 | .sp |
| 902 | The \fBlocale\fP modifier must specify the name of a locale, for example: |
| 903 | .sp |
| 904 | /pattern/locale=fr_FR |
| 905 | .sp |
| 906 | The given locale is set, \fBpcre2_maketables()\fP is called to build a set of |
| 907 | character tables for the locale, and this is then passed to |
| 908 | \fBpcre2_compile()\fP when compiling the regular expression. The same tables |
| 909 | are used when matching the following subject lines. The \fBlocale\fP modifier |
| 910 | applies only to the pattern on which it appears, but can be given in a |
| 911 | \fB#pattern\fP command if a default is needed. Setting a locale and alternate |
| 912 | character tables are mutually exclusive. |
| 913 | . |
| 914 | . |
| 915 | .SS "Showing pattern memory" |
| 916 | .rs |
| 917 | .sp |
| 918 | The \fBmemory\fP modifier causes the size in bytes of the memory used to hold |
| 919 | the compiled pattern to be output. This does not include the size of the |
| 920 | \fBpcre2_code\fP block; it is just the actual compiled data. If the pattern is |
| 921 | subsequently passed to the JIT compiler, the size of the JIT compiled code is |
| 922 | also output. Here is an example: |
| 923 | .sp |
| 924 | re> /a(b)c/jit,memory |
| 925 | Memory allocation (code space): 21 |
| 926 | Memory allocation (JIT code): 1910 |
| 927 | .sp |
| 928 | . |
| 929 | . |
| 930 | .SS "Limiting nested parentheses" |
| 931 | .rs |
| 932 | .sp |
| 933 | The \fBparens_nest_limit\fP modifier sets a limit on the depth of nested |
| 934 | parentheses in a pattern. Breaching the limit causes a compilation error. |
| 935 | The default for the library is set when PCRE2 is built, but \fBpcre2test\fP |
| 936 | sets its own default of 220, which is required for running the standard test |
| 937 | suite. |
| 938 | . |
| 939 | . |
| 940 | .SS "Limiting the pattern length" |
| 941 | .rs |
| 942 | .sp |
| 943 | The \fBmax_pattern_length\fP modifier sets a limit, in code units, to the |
| 944 | length of pattern that \fBpcre2_compile()\fP will accept. Breaching the limit |
| 945 | causes a compilation error. The default is the largest number a PCRE2_SIZE |
| 946 | variable can hold (essentially unlimited). |
| 947 | . |
| 948 | . |
| 949 | .\" HTML <a name="posixwrapper"></a> |
| 950 | .SS "Using the POSIX wrapper API" |
| 951 | .rs |
| 952 | .sp |
| 953 | The \fBposix\fP and \fBposix_nosub\fP modifiers cause \fBpcre2test\fP to call |
| 954 | PCRE2 via the POSIX wrapper API rather than its native API. When |
| 955 | \fBposix_nosub\fP is used, the POSIX option REG_NOSUB is passed to |
| 956 | \fBregcomp()\fP. The POSIX wrapper supports only the 8-bit library. Note that |
| 957 | it does not imply POSIX matching semantics; for more detail see the |
| 958 | .\" HREF |
| 959 | \fBpcre2posix\fP |
| 960 | .\" |
| 961 | documentation. The following pattern modifiers set options for the |
| 962 | \fBregcomp()\fP function: |
| 963 | .sp |
| 964 | caseless REG_ICASE |
| 965 | multiline REG_NEWLINE |
| 966 | dotall REG_DOTALL ) |
| 967 | ungreedy REG_UNGREEDY ) These options are not part of |
| 968 | ucp REG_UCP ) the POSIX standard |
| 969 | utf REG_UTF8 ) |
| 970 | .sp |
| 971 | The \fBregerror_buffsize\fP modifier specifies a size for the error buffer that |
| 972 | is passed to \fBregerror()\fP in the event of a compilation error. For example: |
| 973 | .sp |
| 974 | /abc/posix,regerror_buffsize=20 |
| 975 | .sp |
| 976 | This provides a means of testing the behaviour of \fBregerror()\fP when the |
| 977 | buffer is too small for the error message. If this modifier has not been set, a |
| 978 | large buffer is used. |
| 979 | .P |
| 980 | The \fBaftertext\fP and \fBallaftertext\fP subject modifiers work as described |
| 981 | below. All other modifiers are either ignored, with a warning message, or cause |
| 982 | an error. |
| 983 | .P |
| 984 | The pattern is passed to \fBregcomp()\fP as a zero-terminated string by |
| 985 | default, but if the \fBuse_length\fP or \fBhex\fP modifiers are set, the |
| 986 | REG_PEND extension is used to pass it by length. |
| 987 | . |
| 988 | . |
| 989 | .SS "Testing the stack guard feature" |
| 990 | .rs |
| 991 | .sp |
| 992 | The \fBstackguard\fP modifier is used to test the use of |
| 993 | \fBpcre2_set_compile_recursion_guard()\fP, a function that is provided to |
| 994 | enable stack availability to be checked during compilation (see the |
| 995 | .\" HREF |
| 996 | \fBpcre2api\fP |
| 997 | .\" |
| 998 | documentation for details). If the number specified by the modifier is greater |
| 999 | than zero, \fBpcre2_set_compile_recursion_guard()\fP is called to set up |
| 1000 | callback from \fBpcre2_compile()\fP to a local function. The argument it |
| 1001 | receives is the current nesting parenthesis depth; if this is greater than the |
| 1002 | value given by the modifier, non-zero is returned, causing the compilation to |
| 1003 | be aborted. |
| 1004 | . |
| 1005 | . |
| 1006 | .SS "Using alternative character tables" |
| 1007 | .rs |
| 1008 | .sp |
| 1009 | The value specified for the \fBtables\fP modifier must be one of the digits 0, |
| 1010 | 1, 2, or 3. It causes a specific set of built-in character tables to be passed |
| 1011 | to \fBpcre2_compile()\fP. This is used in the PCRE2 tests to check behaviour |
| 1012 | with different character tables. The digit specifies the tables as follows: |
| 1013 | .sp |
| 1014 | 0 do not pass any special character tables |
| 1015 | 1 the default ASCII tables, as distributed in |
| 1016 | pcre2_chartables.c.dist |
| 1017 | 2 a set of tables defining ISO 8859 characters |
| 1018 | 3 a set of tables loaded by the #loadtables command |
| 1019 | .sp |
| 1020 | In tables 2, some characters whose codes are greater than 128 are identified as |
| 1021 | letters, digits, spaces, etc. Tables 3 can be used only after a |
| 1022 | \fB#loadtables\fP command has loaded them from a binary file. Setting alternate |
| 1023 | character tables and a locale are mutually exclusive. |
| 1024 | . |
| 1025 | . |
| 1026 | .SS "Setting certain match controls" |
| 1027 | .rs |
| 1028 | .sp |
| 1029 | The following modifiers are really subject modifiers, and are described under |
| 1030 | "Subject Modifiers" below. However, they may be included in a pattern's |
| 1031 | modifier list, in which case they are applied to every subject line that is |
| 1032 | processed with that pattern. These modifiers do not affect the compilation |
| 1033 | process. |
| 1034 | .sp |
| 1035 | aftertext show text after match |
| 1036 | allaftertext show text after captures |
| 1037 | allcaptures show all captures |
| 1038 | allvector show the entire ovector |
| 1039 | allusedtext show all consulted text |
| 1040 | altglobal alternative global matching |
| 1041 | /g global global matching |
| 1042 | jitstack=<n> set size of JIT stack |
| 1043 | mark show mark values |
| 1044 | replace=<string> specify a replacement string |
| 1045 | startchar show starting character when relevant |
| 1046 | substitute_callout use substitution callouts |
| 1047 | substitute_extended use PCRE2_SUBSTITUTE_EXTENDED |
| 1048 | substitute_literal use PCRE2_SUBSTITUTE_LITERAL |
| 1049 | substitute_matched use PCRE2_SUBSTITUTE_MATCHED |
| 1050 | substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH |
| 1051 | substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY |
| 1052 | substitute_skip=<n> skip substitution <n> |
| 1053 | substitute_stop=<n> skip substitution <n> and following |
| 1054 | substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET |
| 1055 | substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY |
| 1056 | .sp |
| 1057 | These modifiers may not appear in a \fB#pattern\fP command. If you want them as |
| 1058 | defaults, set them in a \fB#subject\fP command. |
| 1059 | . |
| 1060 | . |
| 1061 | .SS "Specifying literal subject lines" |
| 1062 | .rs |
| 1063 | .sp |
| 1064 | If the \fBsubject_literal\fP modifier is present on a pattern, all the subject |
| 1065 | lines that it matches are taken as literal strings, with no interpretation of |
| 1066 | backslashes. It is not possible to set subject modifiers on such lines, but any |
| 1067 | that are set as defaults by a \fB#subject\fP command are recognized. |
| 1068 | . |
| 1069 | . |
| 1070 | .SS "Saving a compiled pattern" |
| 1071 | .rs |
| 1072 | .sp |
| 1073 | When a pattern with the \fBpush\fP modifier is successfully compiled, it is |
| 1074 | pushed onto a stack of compiled patterns, and \fBpcre2test\fP expects the next |
| 1075 | line to contain a new pattern (or a command) instead of a subject line. This |
| 1076 | facility is used when saving compiled patterns to a file, as described in the |
| 1077 | section entitled "Saving and restoring compiled patterns" |
| 1078 | .\" HTML <a href="#saverestore"> |
| 1079 | .\" </a> |
| 1080 | below. |
| 1081 | .\" |
| 1082 | If \fBpushcopy\fP is used instead of \fBpush\fP, a copy of the compiled |
| 1083 | pattern is stacked, leaving the original as current, ready to match the |
| 1084 | following input lines. This provides a way of testing the |
| 1085 | \fBpcre2_code_copy()\fP function. |
| 1086 | .\" |
| 1087 | The \fBpush\fP and \fBpushcopy \fP modifiers are incompatible with compilation |
| 1088 | modifiers such as \fBglobal\fP that act at match time. Any that are specified |
| 1089 | are ignored (for the stacked copy), with a warning message, except for |
| 1090 | \fBreplace\fP, which causes an error. Note that \fBjitverify\fP, which is |
| 1091 | allowed, does not carry through to any subsequent matching that uses a stacked |
| 1092 | pattern. |
| 1093 | . |
| 1094 | . |
| 1095 | .SS "Testing foreign pattern conversion" |
| 1096 | .rs |
| 1097 | .sp |
| 1098 | The experimental foreign pattern conversion functions in PCRE2 can be tested by |
| 1099 | setting the \fBconvert\fP modifier. Its argument is a colon-separated list of |
| 1100 | options, which set the equivalent option for the \fBpcre2_pattern_convert()\fP |
| 1101 | function: |
| 1102 | .sp |
| 1103 | glob PCRE2_CONVERT_GLOB |
| 1104 | glob_no_starstar PCRE2_CONVERT_GLOB_NO_STARSTAR |
| 1105 | glob_no_wild_separator PCRE2_CONVERT_GLOB_NO_WILD_SEPARATOR |
| 1106 | posix_basic PCRE2_CONVERT_POSIX_BASIC |
| 1107 | posix_extended PCRE2_CONVERT_POSIX_EXTENDED |
| 1108 | unset Unset all options |
| 1109 | .sp |
| 1110 | The "unset" value is useful for turning off a default that has been set by a |
| 1111 | \fB#pattern\fP command. When one of these options is set, the input pattern is |
| 1112 | passed to \fBpcre2_pattern_convert()\fP. If the conversion is successful, the |
| 1113 | result is reflected in the output and then passed to \fBpcre2_compile()\fP. The |
| 1114 | normal \fButf\fP and \fBno_utf_check\fP options, if set, cause the |
| 1115 | PCRE2_CONVERT_UTF and PCRE2_CONVERT_NO_UTF_CHECK options to be passed to |
| 1116 | \fBpcre2_pattern_convert()\fP. |
| 1117 | .P |
| 1118 | By default, the conversion function is allowed to allocate a buffer for its |
| 1119 | output. However, if the \fBconvert_length\fP modifier is set to a value greater |
| 1120 | than zero, \fBpcre2test\fP passes a buffer of the given length. This makes it |
| 1121 | possible to test the length check. |
| 1122 | .P |
| 1123 | The \fBconvert_glob_escape\fP and \fBconvert_glob_separator\fP modifiers can be |
| 1124 | used to specify the escape and separator characters for glob processing, |
| 1125 | overriding the defaults, which are operating-system dependent. |
| 1126 | . |
| 1127 | . |
| 1128 | .\" HTML <a name="subjectmodifiers"></a> |
| 1129 | .SH "SUBJECT MODIFIERS" |
| 1130 | .rs |
| 1131 | .sp |
| 1132 | The modifiers that can appear in subject lines and the \fB#subject\fP |
| 1133 | command are of two types. |
| 1134 | . |
| 1135 | . |
| 1136 | .SS "Setting match options" |
| 1137 | .rs |
| 1138 | .sp |
| 1139 | The following modifiers set options for \fBpcre2_match()\fP or |
| 1140 | \fBpcre2_dfa_match()\fP. See |
| 1141 | .\" HREF |
| 1142 | \fBpcreapi\fP |
| 1143 | .\" |
| 1144 | for a description of their effects. |
| 1145 | .sp |
| 1146 | anchored set PCRE2_ANCHORED |
| 1147 | endanchored set PCRE2_ENDANCHORED |
| 1148 | dfa_restart set PCRE2_DFA_RESTART |
| 1149 | dfa_shortest set PCRE2_DFA_SHORTEST |
| 1150 | no_jit set PCRE2_NO_JIT |
| 1151 | no_utf_check set PCRE2_NO_UTF_CHECK |
| 1152 | notbol set PCRE2_NOTBOL |
| 1153 | notempty set PCRE2_NOTEMPTY |
| 1154 | notempty_atstart set PCRE2_NOTEMPTY_ATSTART |
| 1155 | noteol set PCRE2_NOTEOL |
| 1156 | partial_hard (or ph) set PCRE2_PARTIAL_HARD |
| 1157 | partial_soft (or ps) set PCRE2_PARTIAL_SOFT |
| 1158 | .sp |
| 1159 | The partial matching modifiers are provided with abbreviations because they |
| 1160 | appear frequently in tests. |
| 1161 | .P |
| 1162 | If the \fBposix\fP or \fBposix_nosub\fP modifier was present on the pattern, |
| 1163 | causing the POSIX wrapper API to be used, the only option-setting modifiers |
| 1164 | that have any effect are \fBnotbol\fP, \fBnotempty\fP, and \fBnoteol\fP, |
| 1165 | causing REG_NOTBOL, REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to |
| 1166 | \fBregexec()\fP. The other modifiers are ignored, with a warning message. |
| 1167 | .P |
| 1168 | There is one additional modifier that can be used with the POSIX wrapper. It is |
| 1169 | ignored (with a warning) if used for non-POSIX matching. |
| 1170 | .sp |
| 1171 | posix_startend=<n>[:<m>] |
| 1172 | .sp |
| 1173 | This causes the subject string to be passed to \fBregexec()\fP using the |
| 1174 | REG_STARTEND option, which uses offsets to specify which part of the string is |
| 1175 | searched. If only one number is given, the end offset is passed as the end of |
| 1176 | the subject string. For more detail of REG_STARTEND, see the |
| 1177 | .\" HREF |
| 1178 | \fBpcre2posix\fP |
| 1179 | .\" |
| 1180 | documentation. If the subject string contains binary zeros (coded as escapes |
| 1181 | such as \ex{00} because \fBpcre2test\fP does not support actual binary zeros in |
| 1182 | its input), you must use \fBposix_startend\fP to specify its length. |
| 1183 | . |
| 1184 | . |
| 1185 | .SS "Setting match controls" |
| 1186 | .rs |
| 1187 | .sp |
| 1188 | The following modifiers affect the matching process or request additional |
| 1189 | information. Some of them may also be specified on a pattern line (see above), |
| 1190 | in which case they apply to every subject line that is matched against that |
| 1191 | pattern, but can be overridden by modifiers on the subject. |
| 1192 | .sp |
| 1193 | aftertext show text after match |
| 1194 | allaftertext show text after captures |
| 1195 | allcaptures show all captures |
| 1196 | allvector show the entire ovector |
| 1197 | allusedtext show all consulted text (non-JIT only) |
| 1198 | altglobal alternative global matching |
| 1199 | callout_capture show captures at callout time |
| 1200 | callout_data=<n> set a value to pass via callouts |
| 1201 | callout_error=<n>[:<m>] control callout error |
| 1202 | callout_extra show extra callout information |
| 1203 | callout_fail=<n>[:<m>] control callout failure |
| 1204 | callout_no_where do not show position of a callout |
| 1205 | callout_none do not supply a callout function |
| 1206 | copy=<number or name> copy captured substring |
| 1207 | depth_limit=<n> set a depth limit |
| 1208 | dfa use \fBpcre2_dfa_match()\fP |
| 1209 | find_limits find match and depth limits |
| 1210 | get=<number or name> extract captured substring |
| 1211 | getall extract all captured substrings |
| 1212 | /g global global matching |
| 1213 | heap_limit=<n> set a limit on heap memory (Kbytes) |
| 1214 | jitstack=<n> set size of JIT stack |
| 1215 | mark show mark values |
| 1216 | match_limit=<n> set a match limit |
| 1217 | memory show heap memory usage |
| 1218 | null_context match with a NULL context |
Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 1219 | null_replacement substitute with NULL replacement |
| 1220 | null_subject match with NULL subject |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 1221 | offset=<n> set starting offset |
| 1222 | offset_limit=<n> set offset limit |
| 1223 | ovector=<n> set size of output vector |
| 1224 | recursion_limit=<n> obsolete synonym for depth_limit |
| 1225 | replace=<string> specify a replacement string |
| 1226 | startchar show startchar when relevant |
| 1227 | startoffset=<n> same as offset=<n> |
| 1228 | substitute_callout use substitution callouts |
| 1229 | substitute_extedded use PCRE2_SUBSTITUTE_EXTENDED |
| 1230 | substitute_literal use PCRE2_SUBSTITUTE_LITERAL |
| 1231 | substitute_matched use PCRE2_SUBSTITUTE_MATCHED |
| 1232 | substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH |
| 1233 | substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY |
| 1234 | substitute_skip=<n> skip substitution number n |
| 1235 | substitute_stop=<n> skip substitution number n and greater |
| 1236 | substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET |
| 1237 | substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY |
| 1238 | zero_terminate pass the subject as zero-terminated |
| 1239 | .sp |
| 1240 | The effects of these modifiers are described in the following sections. When |
| 1241 | matching via the POSIX wrapper API, the \fBaftertext\fP, \fBallaftertext\fP, |
| 1242 | and \fBovector\fP subject modifiers work as described below. All other |
| 1243 | modifiers are either ignored, with a warning message, or cause an error. |
| 1244 | . |
| 1245 | . |
| 1246 | .SS "Showing more text" |
| 1247 | .rs |
| 1248 | .sp |
| 1249 | The \fBaftertext\fP modifier requests that as well as outputting the part of |
| 1250 | the subject string that matched the entire pattern, \fBpcre2test\fP should in |
| 1251 | addition output the remainder of the subject string. This is useful for tests |
| 1252 | where the subject contains multiple copies of the same substring. The |
| 1253 | \fBallaftertext\fP modifier requests the same action for captured substrings as |
| 1254 | well as the main matched substring. In each case the remainder is output on the |
| 1255 | following line with a plus character following the capture number. |
| 1256 | .P |
| 1257 | The \fBallusedtext\fP modifier requests that all the text that was consulted |
| 1258 | during a successful pattern match by the interpreter should be shown, for both |
| 1259 | full and partial matches. This feature is not supported for JIT matching, and |
| 1260 | if requested with JIT it is ignored (with a warning message). Setting this |
| 1261 | modifier affects the output if there is a lookbehind at the start of a match, |
| 1262 | or, for a complete match, a lookahead at the end, or if \eK is used in the |
| 1263 | pattern. Characters that precede or follow the start and end of the actual |
| 1264 | match are indicated in the output by '<' or '>' characters underneath them. |
| 1265 | Here is an example: |
| 1266 | .sp |
| 1267 | re> /(?<=pqr)abc(?=xyz)/ |
| 1268 | data> 123pqrabcxyz456\e=allusedtext |
| 1269 | 0: pqrabcxyz |
| 1270 | <<< >>> |
| 1271 | data> 123pqrabcxy\e=ph,allusedtext |
| 1272 | Partial match: pqrabcxy |
| 1273 | <<< |
| 1274 | .sp |
| 1275 | The first, complete match shows that the matched string is "abc", with the |
| 1276 | preceding and following strings "pqr" and "xyz" having been consulted during |
| 1277 | the match (when processing the assertions). The partial match can indicate only |
| 1278 | the preceding string. |
| 1279 | .P |
| 1280 | The \fBstartchar\fP modifier requests that the starting character for the match |
| 1281 | be indicated, if it is different to the start of the matched string. The only |
| 1282 | time when this occurs is when \eK has been processed as part of the match. In |
| 1283 | this situation, the output for the matched string is displayed from the |
| 1284 | starting character instead of from the match point, with circumflex characters |
| 1285 | under the earlier characters. For example: |
| 1286 | .sp |
| 1287 | re> /abc\eKxyz/ |
| 1288 | data> abcxyz\e=startchar |
| 1289 | 0: abcxyz |
| 1290 | ^^^ |
| 1291 | .sp |
| 1292 | Unlike \fBallusedtext\fP, the \fBstartchar\fP modifier can be used with JIT. |
| 1293 | However, these two modifiers are mutually exclusive. |
| 1294 | . |
| 1295 | . |
| 1296 | .SS "Showing the value of all capture groups" |
| 1297 | .rs |
| 1298 | .sp |
| 1299 | The \fBallcaptures\fP modifier requests that the values of all potential |
| 1300 | captured parentheses be output after a match. By default, only those up to the |
| 1301 | highest one actually used in the match are output (corresponding to the return |
| 1302 | code from \fBpcre2_match()\fP). Groups that did not take part in the match |
| 1303 | are output as "<unset>". This modifier is not relevant for DFA matching (which |
| 1304 | does no capturing) and does not apply when \fBreplace\fP is specified; it is |
| 1305 | ignored, with a warning message, if present. |
| 1306 | . |
| 1307 | . |
| 1308 | .SS "Showing the entire ovector, for all outcomes" |
| 1309 | .rs |
| 1310 | .sp |
| 1311 | The \fBallvector\fP modifier requests that the entire ovector be shown, |
| 1312 | whatever the outcome of the match. Compare \fBallcaptures\fP, which shows only |
| 1313 | up to the maximum number of capture groups for the pattern, and then only for a |
| 1314 | successful complete non-DFA match. This modifier, which acts after any match |
| 1315 | result, and also for DFA matching, provides a means of checking that there are |
| 1316 | no unexpected modifications to ovector fields. Before each match attempt, the |
| 1317 | ovector is filled with a special value, and if this is found in both elements |
| 1318 | of a capturing pair, "<unchanged>" is output. After a successful match, this |
| 1319 | applies to all groups after the maximum capture group for the pattern. In other |
| 1320 | cases it applies to the entire ovector. After a partial match, the first two |
| 1321 | elements are the only ones that should be set. After a DFA match, the amount of |
| 1322 | ovector that is used depends on the number of matches that were found. |
| 1323 | . |
| 1324 | . |
| 1325 | .SS "Testing pattern callouts" |
| 1326 | .rs |
| 1327 | .sp |
| 1328 | A callout function is supplied when \fBpcre2test\fP calls the library matching |
| 1329 | functions, unless \fBcallout_none\fP is specified. Its behaviour can be |
| 1330 | controlled by various modifiers listed above whose names begin with |
| 1331 | \fBcallout_\fP. Details are given in the section entitled "Callouts" |
| 1332 | .\" HTML <a href="#callouts"> |
| 1333 | .\" </a> |
| 1334 | below. |
| 1335 | .\" |
| 1336 | Testing callouts from \fBpcre2_substitute()\fP is decribed separately in |
| 1337 | "Testing the substitution function" |
| 1338 | .\" HTML <a href="#substitution"> |
| 1339 | .\" </a> |
| 1340 | below. |
| 1341 | .\" |
| 1342 | . |
| 1343 | . |
| 1344 | .SS "Finding all matches in a string" |
| 1345 | .rs |
| 1346 | .sp |
| 1347 | Searching for all possible matches within a subject can be requested by the |
| 1348 | \fBglobal\fP or \fBaltglobal\fP modifier. After finding a match, the matching |
| 1349 | function is called again to search the remainder of the subject. The difference |
| 1350 | between \fBglobal\fP and \fBaltglobal\fP is that the former uses the |
| 1351 | \fIstart_offset\fP argument to \fBpcre2_match()\fP or \fBpcre2_dfa_match()\fP |
| 1352 | to start searching at a new point within the entire string (which is what Perl |
| 1353 | does), whereas the latter passes over a shortened subject. This makes a |
| 1354 | difference to the matching process if the pattern begins with a lookbehind |
| 1355 | assertion (including \eb or \eB). |
| 1356 | .P |
| 1357 | If an empty string is matched, the next match is done with the |
| 1358 | PCRE2_NOTEMPTY_ATSTART and PCRE2_ANCHORED flags set, in order to search for |
| 1359 | another, non-empty, match at the same point in the subject. If this match |
| 1360 | fails, the start offset is advanced, and the normal match is retried. This |
| 1361 | imitates the way Perl handles such cases when using the \fB/g\fP modifier or |
| 1362 | the \fBsplit()\fP function. Normally, the start offset is advanced by one |
| 1363 | character, but if the newline convention recognizes CRLF as a newline, and the |
| 1364 | current character is CR followed by LF, an advance of two characters occurs. |
| 1365 | . |
| 1366 | . |
| 1367 | .SS "Testing substring extraction functions" |
| 1368 | .rs |
| 1369 | .sp |
| 1370 | The \fBcopy\fP and \fBget\fP modifiers can be used to test the |
| 1371 | \fBpcre2_substring_copy_xxx()\fP and \fBpcre2_substring_get_xxx()\fP functions. |
| 1372 | They can be given more than once, and each can specify a capture group name or |
| 1373 | number, for example: |
| 1374 | .sp |
| 1375 | abcd\e=copy=1,copy=3,get=G1 |
| 1376 | .sp |
| 1377 | If the \fB#subject\fP command is used to set default copy and/or get lists, |
| 1378 | these can be unset by specifying a negative number to cancel all numbered |
| 1379 | groups and an empty name to cancel all named groups. |
| 1380 | .P |
| 1381 | The \fBgetall\fP modifier tests \fBpcre2_substring_list_get()\fP, which |
| 1382 | extracts all captured substrings. |
| 1383 | .P |
| 1384 | If the subject line is successfully matched, the substrings extracted by the |
| 1385 | convenience functions are output with C, G, or L after the string number |
| 1386 | instead of a colon. This is in addition to the normal full list. The string |
| 1387 | length (that is, the return from the extraction function) is given in |
| 1388 | parentheses after each substring, followed by the name when the extraction was |
| 1389 | by name. |
| 1390 | . |
| 1391 | . |
| 1392 | .\" HTML <a name="substitution"></a> |
| 1393 | .SS "Testing the substitution function" |
| 1394 | .rs |
| 1395 | .sp |
| 1396 | If the \fBreplace\fP modifier is set, the \fBpcre2_substitute()\fP function is |
| 1397 | called instead of one of the matching functions (or after one call of |
| 1398 | \fBpcre2_match()\fP in the case of PCRE2_SUBSTITUTE_MATCHED). Note that |
| 1399 | replacement strings cannot contain commas, because a comma signifies the end of |
| 1400 | a modifier. This is not thought to be an issue in a test program. |
| 1401 | .P |
| 1402 | Specifying a completely empty replacement string disables this modifier. |
| 1403 | However, it is possible to specify an empty replacement by providing a buffer |
| 1404 | length, as described below, for an otherwise empty replacement. |
| 1405 | .P |
| 1406 | Unlike subject strings, \fBpcre2test\fP does not process replacement strings |
| 1407 | for escape sequences. In UTF mode, a replacement string is checked to see if it |
| 1408 | is a valid UTF-8 string. If so, it is correctly converted to a UTF string of |
| 1409 | the appropriate code unit width. If it is not a valid UTF-8 string, the |
| 1410 | individual code units are copied directly. This provides a means of passing an |
| 1411 | invalid UTF-8 string for testing purposes. |
| 1412 | .P |
| 1413 | The following modifiers set options (in additional to the normal match options) |
| 1414 | for \fBpcre2_substitute()\fP: |
| 1415 | .sp |
| 1416 | global PCRE2_SUBSTITUTE_GLOBAL |
| 1417 | substitute_extended PCRE2_SUBSTITUTE_EXTENDED |
| 1418 | substitute_literal PCRE2_SUBSTITUTE_LITERAL |
| 1419 | substitute_matched PCRE2_SUBSTITUTE_MATCHED |
| 1420 | substitute_overflow_length PCRE2_SUBSTITUTE_OVERFLOW_LENGTH |
| 1421 | substitute_replacement_only PCRE2_SUBSTITUTE_REPLACEMENT_ONLY |
| 1422 | substitute_unknown_unset PCRE2_SUBSTITUTE_UNKNOWN_UNSET |
| 1423 | substitute_unset_empty PCRE2_SUBSTITUTE_UNSET_EMPTY |
| 1424 | .sp |
| 1425 | See the |
| 1426 | .\" HREF |
| 1427 | \fBpcre2api\fP |
| 1428 | .\" |
| 1429 | documentation for details of these options. |
| 1430 | .P |
| 1431 | After a successful substitution, the modified string is output, preceded by the |
| 1432 | number of replacements. This may be zero if there were no matches. Here is a |
| 1433 | simple example of a substitution test: |
| 1434 | .sp |
| 1435 | /abc/replace=xxx |
| 1436 | =abc=abc= |
| 1437 | 1: =xxx=abc= |
| 1438 | =abc=abc=\e=global |
| 1439 | 2: =xxx=xxx= |
| 1440 | .sp |
| 1441 | Subject and replacement strings should be kept relatively short (fewer than 256 |
| 1442 | characters) for substitution tests, as fixed-size buffers are used. To make it |
| 1443 | easy to test for buffer overflow, if the replacement string starts with a |
| 1444 | number in square brackets, that number is passed to \fBpcre2_substitute()\fP as |
| 1445 | the size of the output buffer, with the replacement string starting at the next |
| 1446 | character. Here is an example that tests the edge case: |
| 1447 | .sp |
| 1448 | /abc/ |
| 1449 | 123abc123\e=replace=[10]XYZ |
| 1450 | 1: 123XYZ123 |
| 1451 | 123abc123\e=replace=[9]XYZ |
| 1452 | Failed: error -47: no more memory |
| 1453 | .sp |
| 1454 | The default action of \fBpcre2_substitute()\fP is to return |
| 1455 | PCRE2_ERROR_NOMEMORY when the output buffer is too small. However, if the |
| 1456 | PCRE2_SUBSTITUTE_OVERFLOW_LENGTH option is set (by using the |
| 1457 | \fBsubstitute_overflow_length\fP modifier), \fBpcre2_substitute()\fP continues |
| 1458 | to go through the motions of matching and substituting (but not doing any |
| 1459 | callouts), in order to compute the size of buffer that is required. When this |
| 1460 | happens, \fBpcre2test\fP shows the required buffer length (which includes space |
| 1461 | for the trailing zero) as part of the error message. For example: |
| 1462 | .sp |
| 1463 | /abc/substitute_overflow_length |
| 1464 | 123abc123\e=replace=[9]XYZ |
| 1465 | Failed: error -47: no more memory: 10 code units are needed |
| 1466 | .sp |
| 1467 | A replacement string is ignored with POSIX and DFA matching. Specifying partial |
| 1468 | matching provokes an error return ("bad option value") from |
| 1469 | \fBpcre2_substitute()\fP. |
| 1470 | . |
| 1471 | . |
| 1472 | .SS "Testing substitute callouts" |
| 1473 | .rs |
| 1474 | .sp |
| 1475 | If the \fBsubstitute_callout\fP modifier is set, a substitution callout |
| 1476 | function is set up. The \fBnull_context\fP modifier must not be set, because |
| 1477 | the address of the callout function is passed in a match context. When the |
| 1478 | callout function is called (after each substitution), details of the the input |
| 1479 | and output strings are output. For example: |
| 1480 | .sp |
| 1481 | /abc/g,replace=<$0>,substitute_callout |
| 1482 | abcdefabcpqr |
| 1483 | 1(1) Old 0 3 "abc" New 0 5 "<abc>" |
| 1484 | 2(1) Old 6 9 "abc" New 8 13 "<abc>" |
| 1485 | 2: <abc>def<abc>pqr |
| 1486 | .sp |
| 1487 | The first number on each callout line is the count of matches. The |
| 1488 | parenthesized number is the number of pairs that are set in the ovector (that |
| 1489 | is, one more than the number of capturing groups that were set). Then are |
| 1490 | listed the offsets of the old substring, its contents, and the same for the |
| 1491 | replacement. |
| 1492 | .P |
| 1493 | By default, the substitution callout function returns zero, which accepts the |
| 1494 | replacement and causes matching to continue if /g was used. Two further |
| 1495 | modifiers can be used to test other return values. If \fBsubstitute_skip\fP is |
| 1496 | set to a value greater than zero the callout function returns +1 for the match |
| 1497 | of that number, and similarly \fBsubstitute_stop\fP returns -1. These cause the |
| 1498 | replacement to be rejected, and -1 causes no further matching to take place. If |
| 1499 | either of them are set, \fBsubstitute_callout\fP is assumed. For example: |
| 1500 | .sp |
| 1501 | /abc/g,replace=<$0>,substitute_skip=1 |
| 1502 | abcdefabcpqr |
| 1503 | 1(1) Old 0 3 "abc" New 0 5 "<abc> SKIPPED" |
| 1504 | 2(1) Old 6 9 "abc" New 6 11 "<abc>" |
| 1505 | 2: abcdef<abc>pqr |
| 1506 | abcdefabcpqr\e=substitute_stop=1 |
| 1507 | 1(1) Old 0 3 "abc" New 0 5 "<abc> STOPPED" |
| 1508 | 1: abcdefabcpqr |
| 1509 | .sp |
| 1510 | If both are set for the same number, stop takes precedence. Only a single skip |
| 1511 | or stop is supported, which is sufficient for testing that the feature works. |
| 1512 | . |
| 1513 | . |
| 1514 | .SS "Setting the JIT stack size" |
| 1515 | .rs |
| 1516 | .sp |
| 1517 | The \fBjitstack\fP modifier provides a way of setting the maximum stack size |
| 1518 | that is used by the just-in-time optimization code. It is ignored if JIT |
| 1519 | optimization is not being used. The value is a number of kibibytes (units of |
| 1520 | 1024 bytes). Setting zero reverts to the default of 32KiB. Providing a stack |
| 1521 | that is larger than the default is necessary only for very complicated |
| 1522 | patterns. If \fBjitstack\fP is set non-zero on a subject line it overrides any |
| 1523 | value that was set on the pattern. |
| 1524 | . |
| 1525 | . |
| 1526 | .SS "Setting heap, match, and depth limits" |
| 1527 | .rs |
| 1528 | .sp |
| 1529 | The \fBheap_limit\fP, \fBmatch_limit\fP, and \fBdepth_limit\fP modifiers set |
| 1530 | the appropriate limits in the match context. These values are ignored when the |
| 1531 | \fBfind_limits\fP modifier is specified. |
| 1532 | . |
| 1533 | . |
| 1534 | .SS "Finding minimum limits" |
| 1535 | .rs |
| 1536 | .sp |
| 1537 | If the \fBfind_limits\fP modifier is present on a subject line, \fBpcre2test\fP |
| 1538 | calls the relevant matching function several times, setting different values in |
| 1539 | the match context via \fBpcre2_set_heap_limit()\fP, |
| 1540 | \fBpcre2_set_match_limit()\fP, or \fBpcre2_set_depth_limit()\fP until it finds |
| 1541 | the minimum values for each parameter that allows the match to complete without |
| 1542 | error. If JIT is being used, only the match limit is relevant. |
| 1543 | .P |
| 1544 | When using this modifier, the pattern should not contain any limit settings |
| 1545 | such as (*LIMIT_MATCH=...) within it. If such a setting is present and is |
| 1546 | lower than the minimum matching value, the minimum value cannot be found |
| 1547 | because \fBpcre2_set_match_limit()\fP etc. are only able to reduce the value of |
| 1548 | an in-pattern limit; they cannot increase it. |
| 1549 | .P |
| 1550 | For non-DFA matching, the minimum \fIdepth_limit\fP number is a measure of how |
| 1551 | much nested backtracking happens (that is, how deeply the pattern's tree is |
| 1552 | searched). In the case of DFA matching, \fIdepth_limit\fP controls the depth of |
| 1553 | recursive calls of the internal function that is used for handling pattern |
| 1554 | recursion, lookaround assertions, and atomic groups. |
| 1555 | .P |
| 1556 | For non-DFA matching, the \fImatch_limit\fP number is a measure of the amount |
| 1557 | of backtracking that takes place, and learning the minimum value can be |
| 1558 | instructive. For most simple matches, the number is quite small, but for |
| 1559 | patterns with very large numbers of matching possibilities, it can become large |
| 1560 | very quickly with increasing length of subject string. In the case of DFA |
| 1561 | matching, \fImatch_limit\fP controls the total number of calls, both recursive |
| 1562 | and non-recursive, to the internal matching function, thus controlling the |
| 1563 | overall amount of computing resource that is used. |
| 1564 | .P |
| 1565 | For both kinds of matching, the \fIheap_limit\fP number, which is in kibibytes |
| 1566 | (units of 1024 bytes), limits the amount of heap memory used for matching. A |
| 1567 | value of zero disables the use of any heap memory; many simple pattern matches |
| 1568 | can be done without using the heap, so zero is not an unreasonable setting. |
| 1569 | . |
| 1570 | . |
| 1571 | .SS "Showing MARK names" |
| 1572 | .rs |
| 1573 | .sp |
| 1574 | .P |
| 1575 | The \fBmark\fP modifier causes the names from backtracking control verbs that |
| 1576 | are returned from calls to \fBpcre2_match()\fP to be displayed. If a mark is |
| 1577 | returned for a match, non-match, or partial match, \fBpcre2test\fP shows it. |
| 1578 | For a match, it is on a line by itself, tagged with "MK:". Otherwise, it |
| 1579 | is added to the non-match message. |
| 1580 | . |
| 1581 | . |
| 1582 | .SS "Showing memory usage" |
| 1583 | .rs |
| 1584 | .sp |
| 1585 | The \fBmemory\fP modifier causes \fBpcre2test\fP to log the sizes of all heap |
| 1586 | memory allocation and freeing calls that occur during a call to |
| 1587 | \fBpcre2_match()\fP or \fBpcre2_dfa_match()\fP. These occur only when a match |
| 1588 | requires a bigger vector than the default for remembering backtracking points |
| 1589 | (\fBpcre2_match()\fP) or for internal workspace (\fBpcre2_dfa_match()\fP). In |
| 1590 | many cases there will be no heap memory used and therefore no additional |
| 1591 | output. No heap memory is allocated during matching with JIT, so in that case |
| 1592 | the \fBmemory\fP modifier never has any effect. For this modifier to work, the |
| 1593 | \fBnull_context\fP modifier must not be set on both the pattern and the |
| 1594 | subject, though it can be set on one or the other. |
| 1595 | . |
| 1596 | . |
| 1597 | .SS "Setting a starting offset" |
| 1598 | .rs |
| 1599 | .sp |
| 1600 | The \fBoffset\fP modifier sets an offset in the subject string at which |
| 1601 | matching starts. Its value is a number of code units, not characters. |
| 1602 | . |
| 1603 | . |
| 1604 | .SS "Setting an offset limit" |
| 1605 | .rs |
| 1606 | .sp |
| 1607 | The \fBoffset_limit\fP modifier sets a limit for unanchored matches. If a match |
| 1608 | cannot be found starting at or before this offset in the subject, a "no match" |
| 1609 | return is given. The data value is a number of code units, not characters. When |
| 1610 | this modifier is used, the \fBuse_offset_limit\fP modifier must have been set |
| 1611 | for the pattern; if not, an error is generated. |
| 1612 | . |
| 1613 | . |
| 1614 | .SS "Setting the size of the output vector" |
| 1615 | .rs |
| 1616 | .sp |
| 1617 | The \fBovector\fP modifier applies only to the subject line in which it |
| 1618 | appears, though of course it can also be used to set a default in a |
| 1619 | \fB#subject\fP command. It specifies the number of pairs of offsets that are |
| 1620 | available for storing matching information. The default is 15. |
| 1621 | .P |
| 1622 | A value of zero is useful when testing the POSIX API because it causes |
| 1623 | \fBregexec()\fP to be called with a NULL capture vector. When not testing the |
| 1624 | POSIX API, a value of zero is used to cause |
| 1625 | \fBpcre2_match_data_create_from_pattern()\fP to be called, in order to create a |
| 1626 | match block of exactly the right size for the pattern. (It is not possible to |
| 1627 | create a match block with a zero-length ovector; there is always at least one |
| 1628 | pair of offsets.) |
| 1629 | . |
| 1630 | . |
| 1631 | .SS "Passing the subject as zero-terminated" |
| 1632 | .rs |
| 1633 | .sp |
| 1634 | By default, the subject string is passed to a native API matching function with |
| 1635 | its correct length. In order to test the facility for passing a zero-terminated |
| 1636 | string, the \fBzero_terminate\fP modifier is provided. It causes the length to |
| 1637 | be passed as PCRE2_ZERO_TERMINATED. When matching via the POSIX interface, |
| 1638 | this modifier is ignored, with a warning. |
| 1639 | .P |
| 1640 | When testing \fBpcre2_substitute()\fP, this modifier also has the effect of |
| 1641 | passing the replacement string as zero-terminated. |
| 1642 | . |
| 1643 | . |
Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 1644 | .SS "Passing a NULL context, subject, or replacement" |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 1645 | .rs |
| 1646 | .sp |
| 1647 | Normally, \fBpcre2test\fP passes a context block to \fBpcre2_match()\fP, |
| 1648 | \fBpcre2_dfa_match()\fP, \fBpcre2_jit_match()\fP or \fBpcre2_substitute()\fP. |
| 1649 | If the \fBnull_context\fP modifier is set, however, NULL is passed. This is for |
| 1650 | testing that the matching and substitution functions behave correctly in this |
| 1651 | case (they use default values). This modifier cannot be used with the |
| 1652 | \fBfind_limits\fP or \fBsubstitute_callout\fP modifiers. |
Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 1653 | .P |
| 1654 | Similarly, for testing purposes, if the \fBnull_subject\fP or |
| 1655 | \fBnull_replacement\fP modifier is set, the subject or replacement string |
| 1656 | pointers are passed as NULL, respectively, to the relevant functions. |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 1657 | . |
| 1658 | . |
| 1659 | .SH "THE ALTERNATIVE MATCHING FUNCTION" |
| 1660 | .rs |
| 1661 | .sp |
| 1662 | By default, \fBpcre2test\fP uses the standard PCRE2 matching function, |
| 1663 | \fBpcre2_match()\fP to match each subject line. PCRE2 also supports an |
| 1664 | alternative matching function, \fBpcre2_dfa_match()\fP, which operates in a |
| 1665 | different way, and has some restrictions. The differences between the two |
| 1666 | functions are described in the |
| 1667 | .\" HREF |
| 1668 | \fBpcre2matching\fP |
| 1669 | .\" |
| 1670 | documentation. |
| 1671 | .P |
| 1672 | If the \fBdfa\fP modifier is set, the alternative matching function is used. |
| 1673 | This function finds all possible matches at a given point in the subject. If, |
| 1674 | however, the \fBdfa_shortest\fP modifier is set, processing stops after the |
| 1675 | first match is found. This is always the shortest possible match. |
| 1676 | . |
| 1677 | . |
| 1678 | .SH "DEFAULT OUTPUT FROM pcre2test" |
| 1679 | .rs |
| 1680 | .sp |
| 1681 | This section describes the output when the normal matching function, |
| 1682 | \fBpcre2_match()\fP, is being used. |
| 1683 | .P |
| 1684 | When a match succeeds, \fBpcre2test\fP outputs the list of captured substrings, |
| 1685 | starting with number 0 for the string that matched the whole pattern. |
| 1686 | Otherwise, it outputs "No match" when the return is PCRE2_ERROR_NOMATCH, or |
| 1687 | "Partial match:" followed by the partially matching substring when the |
| 1688 | return is PCRE2_ERROR_PARTIAL. (Note that this is the |
| 1689 | entire substring that was inspected during the partial match; it may include |
| 1690 | characters before the actual match start if a lookbehind assertion, \eK, \eb, |
| 1691 | or \eB was involved.) |
| 1692 | .P |
| 1693 | For any other return, \fBpcre2test\fP outputs the PCRE2 negative error number |
| 1694 | and a short descriptive phrase. If the error is a failed UTF string check, the |
| 1695 | code unit offset of the start of the failing character is also output. Here is |
| 1696 | an example of an interactive \fBpcre2test\fP run. |
| 1697 | .sp |
| 1698 | $ pcre2test |
| 1699 | PCRE2 version 10.22 2016-07-29 |
| 1700 | .sp |
| 1701 | re> /^abc(\ed+)/ |
| 1702 | data> abc123 |
| 1703 | 0: abc123 |
| 1704 | 1: 123 |
| 1705 | data> xyz |
| 1706 | No match |
| 1707 | .sp |
| 1708 | Unset capturing substrings that are not followed by one that is set are not |
| 1709 | shown by \fBpcre2test\fP unless the \fBallcaptures\fP modifier is specified. In |
| 1710 | the following example, there are two capturing substrings, but when the first |
| 1711 | data line is matched, the second, unset substring is not shown. An "internal" |
| 1712 | unset substring is shown as "<unset>", as for the second data line. |
| 1713 | .sp |
| 1714 | re> /(a)|(b)/ |
| 1715 | data> a |
| 1716 | 0: a |
| 1717 | 1: a |
| 1718 | data> b |
| 1719 | 0: b |
| 1720 | 1: <unset> |
| 1721 | 2: b |
| 1722 | .sp |
| 1723 | If the strings contain any non-printing characters, they are output as \exhh |
| 1724 | escapes if the value is less than 256 and UTF mode is not set. Otherwise they |
| 1725 | are output as \ex{hh...} escapes. See below for the definition of non-printing |
| 1726 | characters. If the \fBaftertext\fP modifier is set, the output for substring |
| 1727 | 0 is followed by the the rest of the subject string, identified by "0+" like |
| 1728 | this: |
| 1729 | .sp |
| 1730 | re> /cat/aftertext |
| 1731 | data> cataract |
| 1732 | 0: cat |
| 1733 | 0+ aract |
| 1734 | .sp |
| 1735 | If global matching is requested, the results of successive matching attempts |
| 1736 | are output in sequence, like this: |
| 1737 | .sp |
| 1738 | re> /\eBi(\ew\ew)/g |
| 1739 | data> Mississippi |
| 1740 | 0: iss |
| 1741 | 1: ss |
| 1742 | 0: iss |
| 1743 | 1: ss |
| 1744 | 0: ipp |
| 1745 | 1: pp |
| 1746 | .sp |
| 1747 | "No match" is output only if the first match attempt fails. Here is an example |
| 1748 | of a failure message (the offset 4 that is specified by the \fBoffset\fP |
| 1749 | modifier is past the end of the subject string): |
| 1750 | .sp |
| 1751 | re> /xyz/ |
| 1752 | data> xyz\e=offset=4 |
| 1753 | Error -24 (bad offset value) |
| 1754 | .P |
| 1755 | Note that whereas patterns can be continued over several lines (a plain ">" |
| 1756 | prompt is used for continuations), subject lines may not. However newlines can |
| 1757 | be included in a subject by means of the \en escape (or \er, \er\en, etc., |
| 1758 | depending on the newline sequence setting). |
| 1759 | . |
| 1760 | . |
| 1761 | . |
| 1762 | .SH "OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION" |
| 1763 | .rs |
| 1764 | .sp |
| 1765 | When the alternative matching function, \fBpcre2_dfa_match()\fP, is used, the |
| 1766 | output consists of a list of all the matches that start at the first point in |
| 1767 | the subject where there is at least one match. For example: |
| 1768 | .sp |
| 1769 | re> /(tang|tangerine|tan)/ |
| 1770 | data> yellow tangerine\e=dfa |
| 1771 | 0: tangerine |
| 1772 | 1: tang |
| 1773 | 2: tan |
| 1774 | .sp |
| 1775 | Using the normal matching function on this data finds only "tang". The |
| 1776 | longest matching string is always given first (and numbered zero). After a |
| 1777 | PCRE2_ERROR_PARTIAL return, the output is "Partial match:", followed by the |
| 1778 | partially matching substring. Note that this is the entire substring that was |
| 1779 | inspected during the partial match; it may include characters before the actual |
| 1780 | match start if a lookbehind assertion, \eb, or \eB was involved. (\eK is not |
| 1781 | supported for DFA matching.) |
| 1782 | .P |
| 1783 | If global matching is requested, the search for further matches resumes |
| 1784 | at the end of the longest match. For example: |
| 1785 | .sp |
| 1786 | re> /(tang|tangerine|tan)/g |
| 1787 | data> yellow tangerine and tangy sultana\e=dfa |
| 1788 | 0: tangerine |
| 1789 | 1: tang |
| 1790 | 2: tan |
| 1791 | 0: tang |
| 1792 | 1: tan |
| 1793 | 0: tan |
| 1794 | .sp |
| 1795 | The alternative matching function does not support substring capture, so the |
| 1796 | modifiers that are concerned with captured substrings are not relevant. |
| 1797 | . |
| 1798 | . |
| 1799 | .SH "RESTARTING AFTER A PARTIAL MATCH" |
| 1800 | .rs |
| 1801 | .sp |
| 1802 | When the alternative matching function has given the PCRE2_ERROR_PARTIAL |
| 1803 | return, indicating that the subject partially matched the pattern, you can |
| 1804 | restart the match with additional subject data by means of the |
| 1805 | \fBdfa_restart\fP modifier. For example: |
| 1806 | .sp |
| 1807 | re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/ |
| 1808 | data> 23ja\e=ps,dfa |
| 1809 | Partial match: 23ja |
| 1810 | data> n05\e=dfa,dfa_restart |
| 1811 | 0: n05 |
| 1812 | .sp |
| 1813 | For further information about partial matching, see the |
| 1814 | .\" HREF |
| 1815 | \fBpcre2partial\fP |
| 1816 | .\" |
| 1817 | documentation. |
| 1818 | . |
| 1819 | . |
| 1820 | .\" HTML <a name="callouts"></a> |
| 1821 | .SH CALLOUTS |
| 1822 | .rs |
| 1823 | .sp |
| 1824 | If the pattern contains any callout requests, \fBpcre2test\fP's callout |
| 1825 | function is called during matching unless \fBcallout_none\fP is specified. This |
| 1826 | works with both matching functions, and with JIT, though there are some |
| 1827 | differences in behaviour. The output for callouts with numerical arguments and |
| 1828 | those with string arguments is slightly different. |
| 1829 | . |
| 1830 | . |
| 1831 | .SS "Callouts with numerical arguments" |
| 1832 | .rs |
| 1833 | .sp |
| 1834 | By default, the callout function displays the callout number, the start and |
| 1835 | current positions in the subject text at the callout time, and the next pattern |
| 1836 | item to be tested. For example: |
| 1837 | .sp |
| 1838 | --->pqrabcdef |
| 1839 | 0 ^ ^ \ed |
| 1840 | .sp |
| 1841 | This output indicates that callout number 0 occurred for a match attempt |
| 1842 | starting at the fourth character of the subject string, when the pointer was at |
| 1843 | the seventh character, and when the next pattern item was \ed. Just |
| 1844 | one circumflex is output if the start and current positions are the same, or if |
| 1845 | the current position precedes the start position, which can happen if the |
| 1846 | callout is in a lookbehind assertion. |
| 1847 | .P |
| 1848 | Callouts numbered 255 are assumed to be automatic callouts, inserted as a |
| 1849 | result of the \fBauto_callout\fP pattern modifier. In this case, instead of |
| 1850 | showing the callout number, the offset in the pattern, preceded by a plus, is |
| 1851 | output. For example: |
| 1852 | .sp |
| 1853 | re> /\ed?[A-E]\e*/auto_callout |
| 1854 | data> E* |
| 1855 | --->E* |
| 1856 | +0 ^ \ed? |
| 1857 | +3 ^ [A-E] |
| 1858 | +8 ^^ \e* |
| 1859 | +10 ^ ^ |
| 1860 | 0: E* |
| 1861 | .sp |
| 1862 | If a pattern contains (*MARK) items, an additional line is output whenever |
| 1863 | a change of latest mark is passed to the callout function. For example: |
| 1864 | .sp |
| 1865 | re> /a(*MARK:X)bc/auto_callout |
| 1866 | data> abc |
| 1867 | --->abc |
| 1868 | +0 ^ a |
| 1869 | +1 ^^ (*MARK:X) |
| 1870 | +10 ^^ b |
| 1871 | Latest Mark: X |
| 1872 | +11 ^ ^ c |
| 1873 | +12 ^ ^ |
| 1874 | 0: abc |
| 1875 | .sp |
| 1876 | The mark changes between matching "a" and "b", but stays the same for the rest |
| 1877 | of the match, so nothing more is output. If, as a result of backtracking, the |
| 1878 | mark reverts to being unset, the text "<unset>" is output. |
| 1879 | . |
| 1880 | . |
| 1881 | .SS "Callouts with string arguments" |
| 1882 | .rs |
| 1883 | .sp |
| 1884 | The output for a callout with a string argument is similar, except that instead |
| 1885 | of outputting a callout number before the position indicators, the callout |
| 1886 | string and its offset in the pattern string are output before the reflection of |
| 1887 | the subject string, and the subject string is reflected for each callout. For |
| 1888 | example: |
| 1889 | .sp |
| 1890 | re> /^ab(?C'first')cd(?C"second")ef/ |
| 1891 | data> abcdefg |
| 1892 | Callout (7): 'first' |
| 1893 | --->abcdefg |
| 1894 | ^ ^ c |
| 1895 | Callout (20): "second" |
| 1896 | --->abcdefg |
| 1897 | ^ ^ e |
| 1898 | 0: abcdef |
| 1899 | .sp |
| 1900 | . |
| 1901 | . |
| 1902 | .SS "Callout modifiers" |
| 1903 | .rs |
| 1904 | .sp |
| 1905 | The callout function in \fBpcre2test\fP returns zero (carry on matching) by |
| 1906 | default, but you can use a \fBcallout_fail\fP modifier in a subject line to |
| 1907 | change this and other parameters of the callout (see below). |
| 1908 | .P |
| 1909 | If the \fBcallout_capture\fP modifier is set, the current captured groups are |
| 1910 | output when a callout occurs. This is useful only for non-DFA matching, as |
| 1911 | \fBpcre2_dfa_match()\fP does not support capturing, so no captures are ever |
| 1912 | shown. |
| 1913 | .P |
| 1914 | The normal callout output, showing the callout number or pattern offset (as |
| 1915 | described above) is suppressed if the \fBcallout_no_where\fP modifier is set. |
| 1916 | .P |
| 1917 | When using the interpretive matching function \fBpcre2_match()\fP without JIT, |
| 1918 | setting the \fBcallout_extra\fP modifier causes additional output from |
| 1919 | \fBpcre2test\fP's callout function to be generated. For the first callout in a |
| 1920 | match attempt at a new starting position in the subject, "New match attempt" is |
| 1921 | output. If there has been a backtrack since the last callout (or start of |
| 1922 | matching if this is the first callout), "Backtrack" is output, followed by "No |
| 1923 | other matching paths" if the backtrack ended the previous match attempt. For |
| 1924 | example: |
| 1925 | .sp |
| 1926 | re> /(a+)b/auto_callout,no_start_optimize,no_auto_possess |
| 1927 | data> aac\e=callout_extra |
| 1928 | New match attempt |
| 1929 | --->aac |
| 1930 | +0 ^ ( |
| 1931 | +1 ^ a+ |
| 1932 | +3 ^ ^ ) |
| 1933 | +4 ^ ^ b |
| 1934 | Backtrack |
| 1935 | --->aac |
| 1936 | +3 ^^ ) |
| 1937 | +4 ^^ b |
| 1938 | Backtrack |
| 1939 | No other matching paths |
| 1940 | New match attempt |
| 1941 | --->aac |
| 1942 | +0 ^ ( |
| 1943 | +1 ^ a+ |
| 1944 | +3 ^^ ) |
| 1945 | +4 ^^ b |
| 1946 | Backtrack |
| 1947 | No other matching paths |
| 1948 | New match attempt |
| 1949 | --->aac |
| 1950 | +0 ^ ( |
| 1951 | +1 ^ a+ |
| 1952 | Backtrack |
| 1953 | No other matching paths |
| 1954 | New match attempt |
| 1955 | --->aac |
| 1956 | +0 ^ ( |
| 1957 | +1 ^ a+ |
| 1958 | No match |
| 1959 | .sp |
| 1960 | Notice that various optimizations must be turned off if you want all possible |
| 1961 | matching paths to be scanned. If \fBno_start_optimize\fP is not used, there is |
| 1962 | an immediate "no match", without any callouts, because the starting |
| 1963 | optimization fails to find "b" in the subject, which it knows must be present |
| 1964 | for any match. If \fBno_auto_possess\fP is not used, the "a+" item is turned |
| 1965 | into "a++", which reduces the number of backtracks. |
| 1966 | .P |
| 1967 | The \fBcallout_extra\fP modifier has no effect if used with the DFA matching |
| 1968 | function, or with JIT. |
| 1969 | . |
| 1970 | . |
| 1971 | .SS "Return values from callouts" |
| 1972 | .rs |
| 1973 | .sp |
| 1974 | The default return from the callout function is zero, which allows matching to |
| 1975 | continue. The \fBcallout_fail\fP modifier can be given one or two numbers. If |
| 1976 | there is only one number, 1 is returned instead of 0 (causing matching to |
| 1977 | backtrack) when a callout of that number is reached. If two numbers (<n>:<m>) |
| 1978 | are given, 1 is returned when callout <n> is reached and there have been at |
| 1979 | least <m> callouts. The \fBcallout_error\fP modifier is similar, except that |
| 1980 | PCRE2_ERROR_CALLOUT is returned, causing the entire matching process to be |
| 1981 | aborted. If both these modifiers are set for the same callout number, |
| 1982 | \fBcallout_error\fP takes precedence. Note that callouts with string arguments |
| 1983 | are always given the number zero. |
| 1984 | .P |
| 1985 | The \fBcallout_data\fP modifier can be given an unsigned or a negative number. |
| 1986 | This is set as the "user data" that is passed to the matching function, and |
| 1987 | passed back when the callout function is invoked. Any value other than zero is |
| 1988 | used as a return from \fBpcre2test\fP's callout function. |
| 1989 | .P |
| 1990 | Inserting callouts can be helpful when using \fBpcre2test\fP to check |
| 1991 | complicated regular expressions. For further information about callouts, see |
| 1992 | the |
| 1993 | .\" HREF |
| 1994 | \fBpcre2callout\fP |
| 1995 | .\" |
| 1996 | documentation. |
| 1997 | . |
| 1998 | . |
| 1999 | . |
| 2000 | .SH "NON-PRINTING CHARACTERS" |
| 2001 | .rs |
| 2002 | .sp |
| 2003 | When \fBpcre2test\fP is outputting text in the compiled version of a pattern, |
| 2004 | bytes other than 32-126 are always treated as non-printing characters and are |
| 2005 | therefore shown as hex escapes. |
| 2006 | .P |
| 2007 | When \fBpcre2test\fP is outputting text that is a matched part of a subject |
| 2008 | string, it behaves in the same way, unless a different locale has been set for |
| 2009 | the pattern (using the \fBlocale\fP modifier). In this case, the |
| 2010 | \fBisprint()\fP function is used to distinguish printing and non-printing |
| 2011 | characters. |
| 2012 | . |
| 2013 | . |
| 2014 | . |
| 2015 | .\" HTML <a name="saverestore"></a> |
| 2016 | .SH "SAVING AND RESTORING COMPILED PATTERNS" |
| 2017 | .rs |
| 2018 | .sp |
| 2019 | It is possible to save compiled patterns on disc or elsewhere, and reload them |
| 2020 | later, subject to a number of restrictions. JIT data cannot be saved. The host |
| 2021 | on which the patterns are reloaded must be running the same version of PCRE2, |
| 2022 | with the same code unit width, and must also have the same endianness, pointer |
| 2023 | width and PCRE2_SIZE type. Before compiled patterns can be saved they must be |
| 2024 | serialized, that is, converted to a stream of bytes. A single byte stream may |
| 2025 | contain any number of compiled patterns, but they must all use the same |
| 2026 | character tables. A single copy of the tables is included in the byte stream |
| 2027 | (its size is 1088 bytes). |
| 2028 | .P |
| 2029 | The functions whose names begin with \fBpcre2_serialize_\fP are used |
| 2030 | for serializing and de-serializing. They are described in the |
| 2031 | .\" HREF |
| 2032 | \fBpcre2serialize\fP |
| 2033 | .\" |
| 2034 | documentation. In this section we describe the features of \fBpcre2test\fP that |
| 2035 | can be used to test these functions. |
| 2036 | .P |
| 2037 | Note that "serialization" in PCRE2 does not convert compiled patterns to an |
| 2038 | abstract format like Java or .NET. It just makes a reloadable byte code stream. |
| 2039 | Hence the restrictions on reloading mentioned above. |
| 2040 | .P |
| 2041 | In \fBpcre2test\fP, when a pattern with \fBpush\fP modifier is successfully |
| 2042 | compiled, it is pushed onto a stack of compiled patterns, and \fBpcre2test\fP |
| 2043 | expects the next line to contain a new pattern (or command) instead of a |
| 2044 | subject line. By contrast, the \fBpushcopy\fP modifier causes a copy of the |
| 2045 | compiled pattern to be stacked, leaving the original available for immediate |
| 2046 | matching. By using \fBpush\fP and/or \fBpushcopy\fP, a number of patterns can |
| 2047 | be compiled and retained. These modifiers are incompatible with \fBposix\fP, |
| 2048 | and control modifiers that act at match time are ignored (with a message) for |
| 2049 | the stacked patterns. The \fBjitverify\fP modifier applies only at compile |
| 2050 | time. |
| 2051 | .P |
| 2052 | The command |
| 2053 | .sp |
| 2054 | #save <filename> |
| 2055 | .sp |
| 2056 | causes all the stacked patterns to be serialized and the result written to the |
| 2057 | named file. Afterwards, all the stacked patterns are freed. The command |
| 2058 | .sp |
| 2059 | #load <filename> |
| 2060 | .sp |
| 2061 | reads the data in the file, and then arranges for it to be de-serialized, with |
| 2062 | the resulting compiled patterns added to the pattern stack. The pattern on the |
| 2063 | top of the stack can be retrieved by the #pop command, which must be followed |
| 2064 | by lines of subjects that are to be matched with the pattern, terminated as |
| 2065 | usual by an empty line or end of file. This command may be followed by a |
| 2066 | modifier list containing only |
| 2067 | .\" HTML <a href="#controlmodifiers"> |
| 2068 | .\" </a> |
| 2069 | control modifiers |
| 2070 | .\" |
| 2071 | that act after a pattern has been compiled. In particular, \fBhex\fP, |
| 2072 | \fBposix\fP, \fBposix_nosub\fP, \fBpush\fP, and \fBpushcopy\fP are not allowed, |
| 2073 | nor are any |
| 2074 | .\" HTML <a href="#optionmodifiers"> |
| 2075 | .\" </a> |
| 2076 | option-setting modifiers. |
| 2077 | .\" |
| 2078 | The JIT modifiers are, however permitted. Here is an example that saves and |
| 2079 | reloads two patterns. |
| 2080 | .sp |
| 2081 | /abc/push |
| 2082 | /xyz/push |
| 2083 | #save tempfile |
| 2084 | #load tempfile |
| 2085 | #pop info |
| 2086 | xyz |
| 2087 | .sp |
| 2088 | #pop jit,bincode |
| 2089 | abc |
| 2090 | .sp |
| 2091 | If \fBjitverify\fP is used with #pop, it does not automatically imply |
| 2092 | \fBjit\fP, which is different behaviour from when it is used on a pattern. |
| 2093 | .P |
| 2094 | The #popcopy command is analagous to the \fBpushcopy\fP modifier in that it |
| 2095 | makes current a copy of the topmost stack pattern, leaving the original still |
| 2096 | on the stack. |
| 2097 | . |
| 2098 | . |
| 2099 | . |
| 2100 | .SH "SEE ALSO" |
| 2101 | .rs |
| 2102 | .sp |
| 2103 | \fBpcre2\fP(3), \fBpcre2api\fP(3), \fBpcre2callout\fP(3), |
| 2104 | \fBpcre2jit\fP, \fBpcre2matching\fP(3), \fBpcre2partial\fP(d), |
| 2105 | \fBpcre2pattern\fP(3), \fBpcre2serialize\fP(3). |
| 2106 | . |
| 2107 | . |
| 2108 | .SH AUTHOR |
| 2109 | .rs |
| 2110 | .sp |
| 2111 | .nf |
| 2112 | Philip Hazel |
| 2113 | Retired from University Computing Service |
| 2114 | Cambridge, England. |
| 2115 | .fi |
| 2116 | . |
| 2117 | . |
| 2118 | .SH REVISION |
| 2119 | .rs |
| 2120 | .sp |
| 2121 | .nf |
Elliott Hughes | 4e19c8e | 2022-04-15 15:11:02 -0700 | [diff] [blame] | 2122 | Last updated: 12 January 2022 |
| 2123 | Copyright (c) 1997-2022 University of Cambridge. |
Elliott Hughes | 5b80804 | 2021-10-01 10:56:10 -0700 | [diff] [blame] | 2124 | .fi |