Janis Danisevskis | 53e448c | 2016-03-31 13:35:25 +0100 | [diff] [blame] | 1 | Change Log for PCRE2 |
| 2 | -------------------- |
| 3 | |
Janis Danisevskis | 8b979b2 | 2016-08-15 16:09:16 +0100 | [diff] [blame] | 4 | |
Elliott Hughes | 653c210 | 2019-01-09 15:41:36 -0800 | [diff] [blame] | 5 | Version 10.32-RC1 10-September-2018 |
| 6 | ----------------------------------- |
| 7 | |
| 8 | 1. When matching using the the REG_STARTEND feature of the POSIX API with a |
| 9 | non-zero starting offset, unset capturing groups with lower numbers than a |
| 10 | group that did capture something were not being correctly returned as "unset" |
| 11 | (that is, with offset values of -1). |
| 12 | |
| 13 | 2. When matching using the POSIX API, pcre2test used to omit listing unset |
| 14 | groups altogether. Now it shows those that come before any actual captures as |
| 15 | "<unset>", as happens for non-POSIX matching. |
| 16 | |
| 17 | 3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only", |
| 18 | whatever the build configuration was. It now correctly says "\R matches all |
| 19 | Unicode newlines" in the default case when --enable-bsr-anycrlf has not been |
| 20 | specified. Similarly, running "pcre2test -C bsr" never produced the result |
| 21 | ANY. |
| 22 | |
| 23 | 4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing |
| 24 | multi-code-unit characters caused bad behaviour and possibly a crash. This |
| 25 | issue was fixed for other kinds of repeat in release 10.20 by change 19, but |
| 26 | repeating character classes were overlooked. |
| 27 | |
| 28 | 5. pcre2grep now supports the inclusion of binary zeros in patterns that are |
| 29 | read from files via the -f option. |
| 30 | |
| 31 | 6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2. |
| 32 | |
| 33 | 7. Added --enable-jit=auto support to configure.ac. |
| 34 | |
| 35 | 8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit |
| 36 | modes for the benefit of m68k, where pointers can be 16-bit aligned. The |
| 37 | dummies force 32-bit alignment and this ensures that the structure is a |
| 38 | multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other |
| 39 | architectures, alignment requirements take care of this automatically. |
| 40 | |
| 41 | 9. When returning an error from pcre2_pattern_convert(), ensure the error |
| 42 | offset is set zero for early errors. |
| 43 | |
| 44 | 10. A number of patches for Windows support from Daniel Richard G: |
| 45 | |
| 46 | (a) List of error numbers in Runtest.bat corrected (it was not the same as in |
| 47 | Runtest). |
| 48 | |
| 49 | (b) pcre2grep snprintf() workaround as used elsewhere in the tree. |
| 50 | |
| 51 | (c) Support for non-C99 snprintf() that returns -1 in the overflow case. |
| 52 | |
| 53 | 11. Minor tidy of pcre2_dfa_match() code. |
| 54 | |
| 55 | 12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer |
| 56 | use the stack for local workspace and local ovectors. Instead, an initial block |
| 57 | of stack is reserved, but if this is insufficient, heap memory is used. The |
| 58 | heap limit parameter now applies to pcre2_dfa_match(). |
| 59 | |
| 60 | 13. If a "find limits" test of DFA matching in pcre2test resulted in too many |
| 61 | matches for the ovector, no matches were displayed. |
| 62 | |
| 63 | 14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as |
| 64 | EOF. The test looks to have come from a fuzzer. |
| 65 | |
| 66 | 15. If PCRE2 was built with a default match limit a lot greater than the |
| 67 | default default of 10 000 000, some JIT tests of the match limit no longer |
| 68 | failed. All such tests now set 10 000 000 as the upper limit. |
| 69 | |
| 70 | 16. Another Windows related patch for pcregrep to ensure that WIN32 is |
| 71 | undefined under Cygwin. |
| 72 | |
| 73 | 17. Test for the presence of stdint.h and inttypes.h in configure and CMake and |
| 74 | include whichever exists (stdint preferred) instead of unconditionally |
| 75 | including stdint. This makes life easier for old and non-standard systems. |
| 76 | |
| 77 | 18. Further changes to improve portability, especially to old and or non- |
| 78 | standard systems: |
| 79 | |
| 80 | (a) Put all printf arguments in RunGrepTest into single, not double, quotes, |
| 81 | and use \0 not \x00 for binary zero. |
| 82 | |
| 83 | (b) Avoid the use of C++ (i.e. BCPL) // comments. |
| 84 | |
| 85 | (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of |
| 86 | these now, if using MSVC or a standard C before C99, %lu is used with a |
| 87 | cast if necessary. |
| 88 | |
| 89 | 19. Applied a contributed patch to CMakeLists.txt to increase the stack size |
| 90 | when linking pcre2test with MSVC. This gets rid of a stack overflow error in |
| 91 | the standard set of tests. |
| 92 | |
| 93 | 20. Output a warning in pcre2test when ignoring the "altglobal" modifier when |
| 94 | it is given with the "replace" modifier. |
| 95 | |
| 96 | 21. In both pcre2test and pcre2_substitute(), with global matching, a pattern |
| 97 | that matched an empty string, but never at the starting match offset, was not |
| 98 | handled in a Perl-compatible way. The pattern /(<?=\G.)/ is an example of such |
| 99 | a pattern. Because \G is in a lookbehind assertion, there has to be a |
| 100 | "bumpalong" before there can be a match. The automatic "advance by one |
| 101 | character after an empty string match" rule is therefore inappropriate. A more |
| 102 | complicated algorithm has now been implemented. |
| 103 | |
| 104 | 22. When checking to see if a lookbehind is of fixed length, lookaheads were |
| 105 | correctly ignored, but qualifiers on lookaheads were not being ignored, leading |
| 106 | to an incorrect "lookbehind assertion is not fixed length" error. |
| 107 | |
| 108 | 23. The VERSION condition test was reading fractional PCRE2 version numbers |
| 109 | such as the 04 in 10.04 incorrectly and hence giving wrong results. |
| 110 | |
| 111 | 24. Updated to Unicode version 11.0.0. As well as the usual addition of new |
| 112 | scripts and characters, this involved re-jigging the grapheme break property |
| 113 | algorithm because Unicode has changed the way emojis are handled. |
| 114 | |
| 115 | 25. Fixed an obscure bug that struck when there were two atomic groups not |
| 116 | separated by something with a backtracking point. There could be an incorrect |
| 117 | backtrack into the first of the atomic groups. A complicated example is |
| 118 | /(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP |
| 119 | shouldn't find a MARK (because is in an atomic group), but it did. |
| 120 | |
| 121 | 26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set |
| 122 | a list of modifiers for all subsequent patterns - only those that the script |
| 123 | recognizes are meaningful; (2) #subject lines can be used to set or unset a |
| 124 | default "mark" modifier; (3) Unsupported #command lines give a warning when |
| 125 | they are ignored; (4) Mark data is output only if the "mark" modifier is |
| 126 | present. |
| 127 | |
| 128 | 27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported. |
| 129 | |
| 130 | 28. A (*MARK) name was not being passed back for positive assertions that were |
| 131 | terminated by (*ACCEPT). |
| 132 | |
| 133 | 29. Add support for \N{U+dddd}, but only in Unicode mode. |
| 134 | |
| 135 | 30. Add support for (?^) for unsetting all imnsx options. |
| 136 | |
| 137 | 31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose |
| 138 | code point was less than 256 and that were recognized by the lookup table |
| 139 | generated by pcre2_maketables(), which uses isspace() to identify white space. |
| 140 | Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085, |
| 141 | U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by |
| 142 | Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl. |
| 143 | |
| 144 | 32. In certain circumstances, option settings within patterns were not being |
| 145 | correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly |
| 146 | matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the |
| 147 | end of its group during the parse process, but without another setting such as |
| 148 | (?m) the compile phase got it right.) This bug was introduced by the |
| 149 | refactoring in release 10.23. |
| 150 | |
| 151 | 33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to |
| 152 | define memmove() as function call to bcopy(). This hasn't been tested for a |
| 153 | long time because in pcre2test the result of memmove() was being used, whereas |
| 154 | bcopy() doesn't return a result. This feature is now refactored always to call |
| 155 | an emulation function when there is no memmove(). The emulation makes use of |
| 156 | bcopy() when available. |
| 157 | |
| 158 | 34. When serializing a pattern, set the memctl, executable_jit, and tables |
| 159 | fields (that is, all the fields that contain pointers) to zeros so that the |
| 160 | result of serializing is always the same. These fields are re-set when the |
| 161 | pattern is deserialized. |
| 162 | |
| 163 | 35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated |
| 164 | negative class with no characters less than 0x100 followed by a positive class |
| 165 | with only characters less than 0x100, the first class was incorrectly being |
| 166 | auto-possessified, causing incorrect match failures. |
| 167 | |
| 168 | 36. Removed the character type bit ctype_meta, which dates from PCRE1 and is |
| 169 | not used in PCRE2. |
| 170 | |
| 171 | 37. Tidied up unnecessarily complicated macros used in the escapes table. |
| 172 | |
| 173 | 38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted |
| 174 | from distribution tarballs, owing to a typo in Makefile.am which had |
| 175 | testoutput8-16-3 twice. Now fixed. |
| 176 | |
| 177 | 39. If the only branch in a conditional subpattern was anchored, the whole |
| 178 | subpattern was treated as anchored, when it should not have been, since the |
| 179 | assumed empty second branch cannot be anchored. Demonstrated by test patterns |
| 180 | such as /(?(1)^())b/ or /(?(?=^))b/. |
| 181 | |
| 182 | 40. A repeated conditional subpattern that could match an empty string was |
| 183 | always assumed to be unanchored. Now it it checked just like any other |
| 184 | repeated conditional subpattern, and can be found to be anchored if the minimum |
| 185 | quantifier is one or more. I can't see much use for a repeated anchored |
| 186 | pattern, but the behaviour is now consistent. |
| 187 | |
| 188 | 41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint |
| 189 | (for an event that could never occur but you had to have external information |
| 190 | to know that). |
| 191 | |
| 192 | 42. If before the first match in a file that was being searched by pcre2grep |
| 193 | there was a line that was sufficiently long to cause the input buffer to be |
| 194 | expanded, the variable holding the location of the end of the previous match |
| 195 | was being adjusted incorrectly, and could cause an overflow warning from a code |
| 196 | sanitizer. However, as the value is used only to print pending "after" lines |
| 197 | when the next match is reached (and there are no such lines in this case) this |
| 198 | bug could do no damage. |
| 199 | |
| 200 | |
Elliott Hughes | 9bc971b | 2018-07-27 13:23:14 -0700 | [diff] [blame] | 201 | Version 10.31 12-February-2018 |
| 202 | ------------------------------ |
| 203 | |
| 204 | 1. Fix typo (missing ]) in VMS code in pcre2test.c. |
| 205 | |
| 206 | 2. Replace the replicated code for matching extended Unicode grapheme sequences |
| 207 | (which got a lot more complicated by change 10.30/49) by a single subroutine |
| 208 | that is called by both pcre2_match() and pcre2_dfa_match(). |
| 209 | |
| 210 | 3. Add idempotent guard to pcre2_internal.h. |
| 211 | |
| 212 | 4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and |
| 213 | PCRE2_CONFIG_COMPILED_WIDTHS. |
| 214 | |
| 215 | 5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is |
| 216 | defined (e.g. by --enable-never-backslash-C). |
| 217 | |
| 218 | 6. Defined public names for all the pcre2_compile() error numbers, and used |
| 219 | the public names in pcre2_convert.c. |
| 220 | |
| 221 | 7. Fixed a small memory leak in pcre2test (convert contexts). |
| 222 | |
| 223 | 8. Added two casts to compile.c and one to match.c to avoid compiler warnings. |
| 224 | |
| 225 | 9. Added code to pcre2grep when compiled under VMS to set the symbol |
| 226 | PCRE2GREP_RC to the exit status, because VMS does not distinguish between |
| 227 | exit(0) and exit(1). |
| 228 | |
| 229 | 10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain |
| 230 | about a bad option only if the following argument item does not start with a |
| 231 | hyphen. |
| 232 | |
| 233 | 11. pcre2grep was truncating components of file names to 128 characters when |
| 234 | processing files with the -r option, and also (some very odd code) truncating |
| 235 | path names to 512 characters. There is now a check on the absolute length of |
| 236 | full path file names, which may be up to 2047 characters long. |
| 237 | |
| 238 | 12. When an assertion contained (*ACCEPT) it caused all open capturing groups |
| 239 | to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to |
| 240 | misbehaviour for subsequent references to groups that started outside the |
| 241 | assertion. ACCEPT in an assertion now closes only those groups that were |
| 242 | started within that assertion. Fixes oss-fuzz issues 3852 and 3891. |
| 243 | |
| 244 | 13. Multiline matching in pcre2grep was misbehaving if the pattern matched |
| 245 | within a line, and then matched again at the end of the line and over into |
| 246 | subsequent lines. Behaviour was different with and without colouring, and |
| 247 | sometimes context lines were incorrectly printed and/or line endings were lost. |
| 248 | All these issues should now be fixed. |
| 249 | |
| 250 | 14. If --line-buffered was specified for pcre2grep when input was from a |
| 251 | compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be |
| 252 | ignored for compressed files.) |
| 253 | |
| 254 | 15. Although pcre2_jit_match checks whether the pattern is compiled |
| 255 | in a given mode, it was also expected that at least one mode is available. |
| 256 | This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION |
| 257 | when the pattern is not optimized by JIT at all. |
| 258 | |
| 259 | 16. The line number and related variables such as match counts in pcre2grep |
| 260 | were all int variables, causing overflow when files with more than 2147483647 |
| 261 | lines were processed (assuming 32-bit ints). They have all been changed to |
| 262 | unsigned long ints. |
| 263 | |
| 264 | 17. If a backreference with a minimum repeat count of zero was first in a |
| 265 | pattern, apart from assertions, an incorrect first matching character could be |
| 266 | recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set |
| 267 | as the first character of a match. |
| 268 | |
| 269 | 18. Characters in a leading positive assertion are considered for recording a |
| 270 | first character of a match when the rest of the pattern does not provide one. |
| 271 | However, a character in a non-assertive group within a leading assertion such |
| 272 | as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an |
| 273 | infelicity rather than an outright bug, because it did not affect the result of |
| 274 | a match, just its speed. (In fact, in this case, the starting 'a' was |
| 275 | subsequently picked up in the study.) |
| 276 | |
| 277 | 19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return" |
| 278 | instead of "RRETURN" saves unwinding the backtracks in these cases (only one |
| 279 | didn't). |
| 280 | |
| 281 | 20. Allocate a single callout block on the stack at the start of pcre2_match() |
| 282 | and set its never-changing fields once only. Do the same for pcre2_dfa_match(). |
| 283 | |
| 284 | 21. Save the extra compile options (set in the compile context) with the |
| 285 | compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS |
| 286 | to retrieve them, and update pcre2test to show them. |
| 287 | |
| 288 | 22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new |
| 289 | field callout_flags in callout blocks. The bits are set by pcre2_match(), but |
| 290 | not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts |
| 291 | if the callout_extra subject modifier is set. These bits are provided to help |
| 292 | with tracking how a backtracking match is proceeding. |
| 293 | |
| 294 | 23. Updated the pcre2demo.c demonstration program, which was missing the extra |
| 295 | code for -g that handles the case when \K in an assertion causes the match to |
| 296 | end at the original start point. Also arranged for it to detect when \K causes |
| 297 | the end of a match to be before its start. |
| 298 | |
| 299 | 24. Similar to 23 above, strange things (including loops) could happen in |
| 300 | pcre2grep when \K was used in an assertion when --colour was used or in |
| 301 | multiline mode. The "end at original start point" bug is fixed, and if the end |
| 302 | point is found to be before the start point, they are swapped. |
| 303 | |
| 304 | 25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT |
| 305 | matching (both pcre2_match() and pcre2_dfa_match()) and the matched string |
| 306 | started with the first code unit of a newline sequence, matching failed because |
| 307 | it was not tried at the newline. |
| 308 | |
| 309 | 26. Code for giving up a non-partial match after failing to find a starting |
| 310 | code unit anywhere in the subject was missing when searching for one of a |
| 311 | number of code units (the bitmap case) in both pcre2_match() and |
| 312 | pcre2_dfa_match(). This was a missing optimization rather than a bug. |
| 313 | |
| 314 | 27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a |
| 315 | pointer argument rather than a code unit value. This should not have affected |
| 316 | the generated code. |
| 317 | |
| 318 | 28. The JIT compiler has been updated. |
| 319 | |
| 320 | 29. Avoid pointer overflow for unset captures in pcre2_substring_list_get(). |
| 321 | This could not actually cause a crash because it was always used in a memcpy() |
| 322 | call with zero length. |
| 323 | |
| 324 | 30. Some internal structures have a variable-length ovector[] as their last |
| 325 | element. Their actual memory is obtained dynamically, giving an ovector of |
| 326 | appropriate length. However, they are defined in the structure as |
| 327 | ovector[NUMBER], where NUMBER is large so that array bound checkers don't |
| 328 | grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing |
| 329 | groups, making the ovector larger than this. The number has been increased to |
| 330 | 131072, which allows for the maximum number of captures (65535) plus the |
| 331 | overall match. This fixes oss-fuzz issue 5415. |
| 332 | |
| 333 | 31. Auto-possessification at the end of a capturing group was dependent on what |
| 334 | follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused |
| 335 | incorrect behaviour when the group was called recursively from elsewhere in the |
| 336 | pattern where something different might follow. This bug is an unforseen |
| 337 | consequence of change #1 for 10.30 - the implementation of backtracking into |
| 338 | recursions. Iterators at the ends of capturing groups are no longer considered |
| 339 | for auto-possessification if the pattern contains any recursions. Fixes |
| 340 | Bugzilla #2232. |
| 341 | |
| 342 | |
| 343 | Version 10.30 14-August-2017 |
| 344 | ---------------------------- |
| 345 | |
| 346 | 1. The main interpreter, pcre2_match(), has been refactored into a new version |
| 347 | that does not use recursive function calls (and therefore the stack) for |
| 348 | remembering backtracking positions. This makes --disable-stack-for-recursion a |
| 349 | NOOP. The new implementation allows backtracking into recursive group calls in |
| 350 | patterns, making it more compatible with Perl, and also fixes some other |
| 351 | hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because |
| 352 | the old code had a number of fudges to try to reduce stack usage. It seems to |
| 353 | run no slower than the old code. |
| 354 | |
| 355 | A number of bugs in the refactored code were subsequently fixed during testing |
| 356 | before release, but after the code was made available in the repository. These |
| 357 | bugs were never in fully released code, but are noted here for the record. |
| 358 | |
| 359 | (a) If a pattern had fewer capturing parentheses than the ovector supplied in |
| 360 | the match data block, a memory error (detectable by ASAN) occurred after |
| 361 | a match, because the external block was being set from non-existent |
| 362 | internal ovector fields. Fixes oss-fuzz issue 781. |
| 363 | |
| 364 | (b) A pattern with very many capturing parentheses (when the internal frame |
| 365 | size was greater than the initial frame vector on the stack) caused a |
| 366 | crash. A vector on the heap is now set up at the start of matching if the |
| 367 | vector on the stack is not big enough to handle at least 10 frames. |
| 368 | Fixes oss-fuzz issue 783. |
| 369 | |
| 370 | (c) Handling of (*VERB)s in recursions was wrong in some cases. |
| 371 | |
| 372 | (d) Captures in negative assertions that were used as conditions were not |
| 373 | happening if the assertion matched via (*ACCEPT). |
| 374 | |
| 375 | (e) Mark values were not being passed out of recursions. |
| 376 | |
| 377 | (f) Refactor some code in do_callout() to avoid picky compiler warnings about |
| 378 | negative indices. Fixes oss-fuzz issue 1454. |
| 379 | |
| 380 | (g) Similarly refactor the way the variable length ovector is addressed for |
| 381 | similar reasons. Fixes oss-fuzz issue 1465. |
| 382 | |
| 383 | 2. Now that pcre2_match() no longer uses recursive function calls (see above), |
| 384 | the "match limit recursion" value seems misnamed. It still exists, and limits |
| 385 | the depth of tree that is searched. To avoid future confusion, it has been |
| 386 | renamed as "depth limit" in all relevant places (--with-depth-limit, |
| 387 | (*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still |
| 388 | available for backwards compatibility. |
| 389 | |
| 390 | 3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers: |
| 391 | |
| 392 | (a) Check for malloc failures when getting memory for the ovector (POSIX) or |
| 393 | the match data block (non-POSIX). |
| 394 | |
| 395 | 4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property |
| 396 | for a character with a code point greater than 0x10ffff (the Unicode maximum) |
| 397 | caused a crash. |
| 398 | |
| 399 | 5. If a lookbehind assertion that contained a back reference to a group |
| 400 | appearing later in the pattern was compiled with the PCRE2_ANCHORED option, |
| 401 | undefined actions (often a segmentation fault) could occur, depending on what |
| 402 | other options were set. An example assertion is (?<!\1(abc)) where the |
| 403 | reference \1 precedes the group (abc). This fixes oss-fuzz issue 865. |
| 404 | |
| 405 | 6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for |
| 406 | pcre2test to use it to output the frame size when the "framesize" modifier is |
| 407 | given. |
| 408 | |
| 409 | 7. Reworked the recursive pattern matching in the JIT compiler to follow the |
| 410 | interpreter changes. |
| 411 | |
| 412 | 8. When the zero_terminate modifier was specified on a pcre2test subject line |
| 413 | for global matching, unpredictable things could happen. For example, in UTF-8 |
| 414 | mode, the pattern //g,zero_terminate read random memory when matched against an |
| 415 | empty string with zero_terminate. This was a bug in pcre2test, not the library. |
| 416 | |
| 417 | 9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out |
| 418 | of the section that is compiled when Unix-style directory scanning is |
| 419 | available, and into a new section that is always compiled for Windows. |
| 420 | |
| 421 | 10. In pcre2test, explicitly close the file after an error during serialization |
| 422 | or deserialization (the "load" or "save" commands). |
| 423 | |
| 424 | 11. Fix memory leak in pcre2_serialize_decode() when the input is invalid. |
| 425 | |
| 426 | 12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with |
| 427 | a NULL pattern pointer when Unicode support is available. |
| 428 | |
| 429 | 13. When the 32-bit library was being tested by pcre2test, error messages that |
| 430 | were longer than 64 code units could cause a buffer overflow. This was a bug in |
| 431 | pcre2test. |
| 432 | |
| 433 | 14. The alternative matching function, pcre2_dfa_match() misbehaved if it |
| 434 | encountered a character class with a possessive repeat, for example [a-f]{3}+. |
| 435 | |
| 436 | 15. The depth (formerly recursion) limit now applies to DFA matching (as |
| 437 | of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA |
| 438 | matching to find the minimum value for this limit. |
| 439 | |
| 440 | 16. Since 10.21, if pcre2_match() was called with a null context, default |
| 441 | memory allocation functions were used instead of whatever was used when the |
| 442 | pattern was compiled. |
| 443 | |
| 444 | 17. Changes to the pcre2test "memory" modifier on a subject line. These apply |
| 445 | only to pcre2_match(): |
| 446 | |
| 447 | (a) Warn if null_context is set on both pattern and subject, because the |
| 448 | memory details cannot then be shown. |
| 449 | |
| 450 | (b) Remember (up to a certain number of) memory allocations and their |
| 451 | lengths, and list only the lengths, so as to be system-independent. |
| 452 | (In practice, the new interpreter never has more than 2 blocks allocated |
| 453 | simultaneously.) |
| 454 | |
| 455 | 18. Make pcre2test detect an error return from pcre2_get_error_message(), give |
| 456 | a message, and abandon the run (this would have detected #13 above). |
| 457 | |
| 458 | 19. Implemented PCRE2_ENDANCHORED. |
| 459 | |
| 460 | 20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement |
| 461 | the --output=text (-O) option and the inbuilt callout echo. |
| 462 | |
| 463 | 21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and |
| 464 | single-branch conditions with a false condition (e.g. DEFINE) at the start of a |
| 465 | branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as |
| 466 | anchored. |
| 467 | |
| 468 | 22. Added an explicit limit on the amount of heap used by pcre2_match(), set by |
| 469 | pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the |
| 470 | heap limit along with other pattern information, and to find the minimum when |
| 471 | the find_limits modifier is set. |
| 472 | |
| 473 | 23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled |
| 474 | pattern is set up so as to initialize any padding the compiler might have |
| 475 | included. This avoids valgrind warnings when a compiled pattern is copied, in |
| 476 | particular when it is serialized. |
| 477 | |
| 478 | 24. Remove a redundant line of code left in accidentally a long time ago. |
| 479 | |
| 480 | 25. Remove a duplication typo in pcre2_tables.c |
| 481 | |
| 482 | 26. Correct an incorrect cast in pcre2_valid_utf.c |
| 483 | |
| 484 | 27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the |
| 485 | tests to improve coverage. |
| 486 | |
| 487 | 28. Some fixes/tidies as a result of looking at Coverity Scan output: |
| 488 | |
| 489 | (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c. |
| 490 | (b) Added some casts to avoid "suspicious implicit sign extension". |
| 491 | (c) Resource leaks in pcre2test in rare error cases. |
| 492 | (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge |
| 493 | for checking at compile time that tables are the right size. |
| 494 | (e) Add missing "fall through" comment. |
| 495 | |
| 496 | 29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features. |
| 497 | |
| 498 | 30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this. |
| 499 | |
| 500 | 31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in |
| 501 | pcre2test, a crash could occur. |
| 502 | |
Elliott Hughes | 653c210 | 2019-01-09 15:41:36 -0800 | [diff] [blame] | 503 | 32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so |
| 504 | that all the tests can run with clang's sanitizing options. |
Elliott Hughes | 9bc971b | 2018-07-27 13:23:14 -0700 | [diff] [blame] | 505 | |
| 506 | 33. Implement extra compile options in the compile context and add the first |
| 507 | one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES. |
| 508 | |
| 509 | 34. Implement newline type PCRE2_NEWLINE_NUL. |
| 510 | |
| 511 | 35. A lookbehind assertion that had a zero-length branch caused undefined |
| 512 | behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859. |
| 513 | |
| 514 | 36. The match limit value now also applies to pcre2_dfa_match() as there are |
| 515 | patterns that can use up a lot of resources without necessarily recursing very |
| 516 | deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761. |
| 517 | |
| 518 | 37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL. |
| 519 | |
| 520 | 38. Fix returned offsets from regexec() when REG_STARTEND is used with a |
| 521 | starting offset greater than zero. |
| 522 | |
| 523 | 39. Implement REG_PEND (GNU extension) for the POSIX wrapper. |
| 524 | |
| 525 | 40. Implement the subject_literal modifier in pcre2test, and allow jitstack on |
| 526 | pattern lines. |
| 527 | |
| 528 | 41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC. |
| 529 | |
| 530 | 42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit |
| 531 | of pcre2grep. |
| 532 | |
| 533 | 43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL, |
| 534 | PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs: |
| 535 | |
| 536 | (a) The -F option did not work for fixed strings containing \E. |
| 537 | (b) The -w option did not work for patterns with multiple branches. |
| 538 | |
| 539 | 44. Added configuration options for the SELinux compatible execmem allocator in |
| 540 | JIT. |
| 541 | |
| 542 | 45. Increased the limit for searching for a "must be present" code unit in |
| 543 | subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are |
| 544 | much faster. |
| 545 | |
| 546 | 46. Arrange for anchored patterns to record and use "first code unit" data, |
| 547 | because this can give a fast "no match" without searching for a "required code |
| 548 | unit". Previously only non-anchored patterns did this. |
| 549 | |
| 550 | 47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0. |
| 551 | |
| 552 | 48. Add the callout_no_where modifier to pcre2test. |
| 553 | |
| 554 | 49. Update extended grapheme breaking rules to the latest set that are in |
| 555 | Unicode Standard Annex #29. |
| 556 | |
| 557 | 50. Added experimental foreign pattern conversion facilities |
| 558 | (pcre2_pattern_convert() and friends). |
| 559 | |
| 560 | 51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE |
| 561 | is defined in a system header in cygwin. Also modified some of the #ifdefs in |
| 562 | pcre2grep related to Windows and Cygwin support. |
| 563 | |
| 564 | 52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a |
| 565 | character class is the last character in the class, Perl does not give a |
| 566 | warning. PCRE2 now also treats this as a literal. |
| 567 | |
| 568 | 53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was |
| 569 | not doing so for [\d-X] (and similar escapes), as is documented. |
| 570 | |
| 571 | 54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard. |
| 572 | |
| 573 | 55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in |
| 574 | pcre2_compile() which could never actually trigger (code should have been cut |
| 575 | out when Unicode support is disabled). |
| 576 | |
| 577 | |
| 578 | Version 10.23 14-February-2017 |
| 579 | ------------------------------ |
| 580 | |
| 581 | 1. Extended pcre2test with the utf8_input modifier so that it is able to |
| 582 | generate all possible 16-bit and 32-bit code unit values in non-UTF modes. |
| 583 | |
| 584 | 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without |
| 585 | PCRE2_UCP set, a negative character type such as \D in a positive class should |
| 586 | cause all characters greater than 255 to match, whatever else is in the class. |
| 587 | There was a bug that caused this not to happen if a Unicode property item was |
| 588 | added to such a class, for example [\D\P{Nd}] or [\W\pL]. |
| 589 | |
| 590 | 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax |
| 591 | checking is now done in the pre-pass that identifies capturing groups. This has |
| 592 | reduced the amount of duplication and made the code tidier. While doing this, |
| 593 | some minor bugs and Perl incompatibilities were fixed, including: |
| 594 | |
| 595 | (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead |
| 596 | of giving an invalid quantifier error. |
| 597 | |
| 598 | (b) {0} can now be used after a group in a lookbehind assertion; previously |
| 599 | this caused an "assertion is not fixed length" error. |
| 600 | |
| 601 | (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with |
| 602 | the name "DEFINE" exists. PCRE2 now does likewise. |
| 603 | |
| 604 | (d) A recursion condition test such as (?(R2)...) must now refer to an |
| 605 | existing subpattern. |
| 606 | |
| 607 | (e) A conditional recursion test such as (?(R)...) misbehaved if there was a |
| 608 | group whose name began with "R". |
| 609 | |
| 610 | (f) When testing zero-terminated patterns under valgrind, the terminating |
| 611 | zero is now marked "no access". This catches bugs that would otherwise |
| 612 | show up only with non-zero-terminated patterns. |
| 613 | |
| 614 | (g) A hyphen appearing immediately after a POSIX character class (for example |
| 615 | /[[:ascii:]-z]/) now generates an error. Perl does accept this as a |
| 616 | literal, but gives a warning, so it seems best to fail it in PCRE. |
| 617 | |
| 618 | (h) An empty \Q\E sequence may appear after a callout that precedes an |
| 619 | assertion condition (it is, of course, ignored). |
| 620 | |
| 621 | One effect of the refactoring is that some error numbers and messages have |
| 622 | changed, and the pattern offset given for compiling errors is not always the |
| 623 | right-most character that has been read. In particular, for a variable-length |
| 624 | lookbehind assertion it now points to the start of the assertion. Another |
| 625 | change is that when a callout appears before a group, the "length of next |
| 626 | pattern item" that is passed now just gives the length of the opening |
| 627 | parenthesis item, not the length of the whole group. A length of zero is now |
| 628 | given only for a callout at the end of the pattern. Automatic callouts are no |
| 629 | longer inserted before and after explicit callouts in the pattern. |
| 630 | |
| 631 | A number of bugs in the refactored code were subsequently fixed during testing |
| 632 | before release, but after the code was made available in the repository. Many |
| 633 | of the bugs were discovered by fuzzing testing. Several of them were related to |
| 634 | the change from assuming a zero-terminated pattern (which previously had |
| 635 | required non-zero terminated strings to be copied). These bugs were never in |
| 636 | fully released code, but are noted here for the record. |
| 637 | |
| 638 | (a) An overall recursion such as (?0) inside a lookbehind assertion was not |
| 639 | being diagnosed as an error. |
| 640 | |
| 641 | (b) In utf mode, the length of a *MARK (or other verb) name was being checked |
| 642 | in characters instead of code units, which could lead to bad code being |
| 643 | compiled, leading to unpredictable behaviour. |
| 644 | |
| 645 | (c) In extended /x mode, characters whose code was greater than 255 caused |
| 646 | a lookup outside one of the global tables. A similar bug existed for wide |
| 647 | characters in *VERB names. |
| 648 | |
| 649 | (d) The amount of memory needed for a compiled pattern was miscalculated if a |
| 650 | lookbehind contained more than one toplevel branch and the first branch |
| 651 | was of length zero. |
| 652 | |
| 653 | (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero- |
| 654 | terminated pattern, if a # comment ran on to the end of the pattern, one |
| 655 | or more code units past the end were being read. |
| 656 | |
| 657 | (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g. |
| 658 | "{2,2") could cause reading beyond the pattern. |
| 659 | |
| 660 | (g) When reading a callout string, if the end delimiter was at the end of the |
| 661 | pattern one further code unit was read. |
| 662 | |
| 663 | (h) An unterminated number after \g' could cause reading beyond the pattern. |
| 664 | |
| 665 | (i) An insufficient memory size was being computed for compiling with |
| 666 | PCRE2_AUTO_CALLOUT. |
| 667 | |
| 668 | (j) A conditional group with an assertion condition used more memory than was |
| 669 | allowed for it during parsing, so too many of them could therefore |
| 670 | overrun a buffer. |
| 671 | |
| 672 | (k) If parsing a pattern exactly filled the buffer, the internal test for |
| 673 | overrun did not check when the final META_END item was added. |
| 674 | |
| 675 | (l) If a lookbehind contained a subroutine call, and the called group |
| 676 | contained an option setting such as (?s), and the PCRE2_ANCHORED option |
| 677 | was set, unpredictable behaviour could occur. The underlying bug was |
| 678 | incorrect code and insufficient checking while searching for the end of |
| 679 | the called subroutine in the parsed pattern. |
| 680 | |
| 681 | (m) Quantifiers following (*VERB)s were not being diagnosed as errors. |
| 682 | |
| 683 | (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and |
| 684 | PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour. |
| 685 | |
| 686 | (o) If \Q was preceded by a quantified item, and the following \E was |
| 687 | followed by '?' or '+', and there was at least one literal character |
| 688 | between them, an internal error "unexpected repeat" occurred (example: |
| 689 | /.+\QX\E+/). |
| 690 | |
| 691 | (p) A buffer overflow could occur while sorting the names in the group name |
| 692 | list (depending on the order in which the names were seen). |
| 693 | |
| 694 | (q) A conditional group that started with a callout was not doing the right |
| 695 | check for a following assertion, leading to compiling bad code. Example: |
| 696 | /(?(C'XX))?!XX/ |
| 697 | |
| 698 | (r) If a character whose code point was greater than 0xffff appeared within |
| 699 | a lookbehind that was within another lookbehind, the calculation of the |
| 700 | lookbehind length went wrong and could provoke an internal error. |
| 701 | |
| 702 | (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused |
| 703 | an internal error. Now the hyphen is treated as a literal. |
| 704 | |
| 705 | 4. Back references are now permitted in lookbehind assertions when there are |
| 706 | no duplicated group numbers (that is, (?| has not been used), and, if the |
| 707 | reference is by name, there is only one group of that name. The referenced |
| 708 | group must, of course be of fixed length. |
| 709 | |
| 710 | 5. pcre2test has been upgraded so that, when run under valgrind with valgrind |
| 711 | support enabled, reading past the end of the pattern is detected, both when |
| 712 | compiling and during callout processing. |
| 713 | |
| 714 | 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back |
| 715 | reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does |
| 716 | not recognize this syntax. |
| 717 | |
| 718 | 7. Automatic callouts are no longer generated before and after callouts in the |
| 719 | pattern. |
| 720 | |
| 721 | 8. When pcre2test was outputing information from a callout, the caret indicator |
| 722 | for the current position in the subject line was incorrect if it was after an |
| 723 | escape sequence for a character whose code point was greater than \x{ff}. |
| 724 | |
| 725 | 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be |
| 726 | PCRE2_STATIC_RUNTIME). Fix from David Gaussmann. |
| 727 | |
| 728 | 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer |
| 729 | expansion when long lines are encountered. Original patch by Dmitry |
| 730 | Cherniachenko. |
| 731 | |
| 732 | 11. If pcre2grep was compiled with JIT support, but the library was compiled |
| 733 | without it (something that neither ./configure nor CMake allow, but it can be |
| 734 | done by editing config.h), pcre2grep was giving a JIT error. Now it detects |
| 735 | this situation and does not try to use JIT. |
| 736 | |
| 737 | 12. Added some "const" qualifiers to variables in pcre2grep. |
| 738 | |
| 739 | 13. Added Dmitry Cherniachenko's patch for colouring output in Windows |
| 740 | (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment |
| 741 | variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found. |
| 742 | |
| 743 | 14. Add the -t (grand total) option to pcre2grep. |
| 744 | |
| 745 | 15. A number of bugs have been mended relating to match start-up optimizations |
| 746 | when the first thing in a pattern is a positive lookahead. These all applied |
| 747 | only when PCRE2_NO_START_OPTIMIZE was *not* set: |
| 748 | |
| 749 | (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed |
| 750 | both an initial 'X' and a following 'X'. |
| 751 | (b) Some patterns starting with an assertion that started with .* were |
| 752 | incorrectly optimized as having to match at the start of the subject or |
| 753 | after a newline. There are cases where this is not true, for example, |
| 754 | (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that |
| 755 | start with spaces. Starting .* in an assertion is no longer taken as an |
| 756 | indication of matching at the start (or after a newline). |
| 757 | |
| 758 | 16. The "offset" modifier in pcre2test was not being ignored (as documented) |
| 759 | when the POSIX API was in use. |
| 760 | |
| 761 | 17. Added --enable-fuzz-support to "configure", causing an non-installed |
| 762 | library containing a test function that can be called by fuzzers to be |
| 763 | compiled. A non-installed binary to run the test function locally, called |
| 764 | pcre2fuzzcheck is also compiled. |
| 765 | |
| 766 | 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and |
| 767 | which started with .* inside a positive lookahead was incorrectly being |
| 768 | compiled as implicitly anchored. |
| 769 | |
| 770 | 19. Removed all instances of "register" declarations, as they are considered |
| 771 | obsolete these days and in any case had become very haphazard. |
| 772 | |
| 773 | 20. Add strerror() to pcre2test for failed file opening. |
| 774 | |
| 775 | 21. Make pcre2test -C list valgrind support when it is enabled. |
| 776 | |
| 777 | 22. Add the use_length modifier to pcre2test. |
| 778 | |
| 779 | 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and |
| 780 | 'copy' modifiers. |
| 781 | |
| 782 | 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it |
| 783 | is apparently needed there as well as in the function definitions. (Why did |
| 784 | nobody ask for this in PCRE1?) |
| 785 | |
| 786 | 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to |
| 787 | PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard |
| 788 | compliant and unique. |
| 789 | |
| 790 | 26. pcre2-config --libs-posix was listing -lpcre2posix instead of |
| 791 | -lpcre2-posix. Also, the CMake build process was building the library with the |
| 792 | wrong name. |
| 793 | |
| 794 | 27. In pcre2test, give some offset information for errors in hex patterns. |
| 795 | This uses the C99 formatting sequence %td, except for MSVC which doesn't |
| 796 | support it - %lu is used instead. |
| 797 | |
| 798 | 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to |
| 799 | pcre2test for testing it. |
| 800 | |
| 801 | 29. Fix small memory leak in pcre2test. |
| 802 | |
| 803 | 30. Fix out-of-bounds read for partial matching of /./ against an empty string |
| 804 | when the newline type is CRLF. |
| 805 | |
| 806 | 31. Fix a bug in pcre2test that caused a crash when a locale was set either in |
| 807 | the current pattern or a previous one and a wide character was matched. |
| 808 | |
| 809 | 32. The appearance of \p, \P, or \X in a substitution string when |
| 810 | PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL |
| 811 | dereference). |
| 812 | |
| 813 | 33. If the starting offset was specified as greater than the subject length in |
| 814 | a call to pcre2_substitute() an out-of-bounds memory reference could occur. |
| 815 | |
| 816 | 34. When PCRE2 was compiled to use the heap instead of the stack for recursive |
| 817 | calls to match(), a repeated minimizing caseless back reference, or a |
| 818 | maximizing one where the two cases had different numbers of code units, |
| 819 | followed by a caseful back reference, could lose the caselessness of the first |
| 820 | repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX |
| 821 | but didn't). |
| 822 | |
| 823 | 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum |
| 824 | matching length and just records zero. Typically this happens when there are |
| 825 | too many nested or recursive back references. If the limit was reached in |
| 826 | certain recursive cases it failed to be triggered and an internal error could |
| 827 | be the result. |
| 828 | |
| 829 | 36. The pcre2_dfa_match() function now takes note of the recursion limit for |
| 830 | the internal recursive calls that are used for lookrounds and recursions within |
| 831 | the pattern. |
| 832 | |
| 833 | 37. More refactoring has got rid of the internal could_be_empty_branch() |
| 834 | function (around 400 lines of code, including comments) by keeping track of |
| 835 | could-be-emptiness as the pattern is compiled instead of scanning compiled |
| 836 | groups. (This would have been much harder before the refactoring of #3 above.) |
| 837 | This lifts a restriction on the number of branches in a group (more than about |
| 838 | 1100 would give "pattern is too complicated"). |
| 839 | |
| 840 | 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern |
| 841 | auto_callout". |
| 842 | |
| 843 | 39. In a library with Unicode support, incorrect data was compiled for a |
| 844 | pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide |
| 845 | characters to match (for example, /[\s[:^ascii:]]/). |
| 846 | |
| 847 | 40. The callout_error modifier has been added to pcre2test to make it possible |
| 848 | to return PCRE2_ERROR_CALLOUT from a callout. |
| 849 | |
| 850 | 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of |
| 851 | "<esc>[00m". |
| 852 | |
| 853 | 42. The limit in the auto-possessification code that was intended to catch |
| 854 | overly-complicated patterns and not spend too much time auto-possessifying was |
| 855 | being reset too often, resulting in very long compile times for some patterns. |
| 856 | Now such patterns are no longer completely auto-possessified. |
| 857 | |
| 858 | 43. Applied Jason Hood's revised patch for RunTest.bat. |
| 859 | |
| 860 | 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood. |
| 861 | |
| 862 | 45. Minor cosmetic fix to pcre2test: move a variable that is not used under |
| 863 | Windows into the "not Windows" code. |
| 864 | |
| 865 | 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy |
| 866 | some of the code: |
| 867 | |
| 868 | * normalised the Windows condition by ensuring WIN32 is defined; |
| 869 | * enables the callout feature under Windows; |
| 870 | * adds globbing (Microsoft's implementation expands quoted args), |
| 871 | using a tweaked opendirectory; |
| 872 | * implements the is_*_tty functions for Windows; |
| 873 | * --color=always will write the ANSI sequences to file; |
| 874 | * add sequences 4 (underline works on Win10) and 5 (blink as bright |
| 875 | background, relatively standard on DOS/Win); |
| 876 | * remove the (char *) casts for the now-const strings; |
| 877 | * remove GREP_COLOUR (grep's command line allowed the 'u', but not |
| 878 | the environment), parsing GREP_COLORS instead; |
| 879 | * uses the current colour if not set, rather than black; |
| 880 | * add print_match for the undefined case; |
| 881 | * fixes a typo. |
| 882 | |
| 883 | In addition, colour settings containing anything other than digits and |
| 884 | semicolon are ignored, and the colour controls are no longer output for empty |
| 885 | strings. |
| 886 | |
| 887 | 47. Detecting patterns that are too large inside the length-measuring loop |
| 888 | saves processing ridiculously long patterns to their end. |
| 889 | |
| 890 | 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it |
| 891 | just wastes time. In the UTF case it can also produce redundant entries in |
| 892 | XCLASS lists caused by characters with multiple other cases and pairs of |
| 893 | characters in the same "not-x" sublists. |
| 894 | |
| 895 | 49. A pattern such as /(?=(a\K))/ can report the end of the match being before |
| 896 | its start; pcre2test was not handling this correctly when using the POSIX |
| 897 | interface (it was OK with the native interface). |
| 898 | |
| 899 | 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will |
| 900 | continue to work, falling back to interpretation if anything goes wrong with |
| 901 | JIT. |
| 902 | |
| 903 | 51. Applied patches from Christian Persch to configure.ac to make use of the |
| 904 | AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT |
| 905 | modules. |
| 906 | |
| 907 | 52. Minor fixes to pcre2grep from Jason Hood: |
| 908 | * fixed some spacing; |
| 909 | * Windows doesn't usually use single quotes, so I've added a define |
| 910 | to use appropriate quotes [in an example]; |
| 911 | * LC_ALL was displayed as "LCC_ALL"; |
| 912 | * numbers 11, 12 & 13 should end in "th"; |
| 913 | * use double quotes in usage message. |
| 914 | |
| 915 | 53. When autopossessifying, skip empty branches without recursion, to reduce |
| 916 | stack usage for the benefit of clang with -fsanitize-address, which uses huge |
| 917 | stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553. |
| 918 | |
| 919 | 54. A pattern with very many explicit back references to a group that is a long |
| 920 | way from the start of the pattern could take a long time to compile because |
| 921 | searching for the referenced group in order to find the minimum length was |
| 922 | being done repeatedly. Now up to 128 group minimum lengths are cached and the |
| 923 | attempt to find a minimum length is abandoned if there is a back reference to a |
| 924 | group whose number is greater than 128. (In that case, the pattern is so |
| 925 | complicated that this optimization probably isn't worth it.) This fixes |
| 926 | oss-fuzz issue 557. |
| 927 | |
| 928 | 55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline |
| 929 | mode with --only-matching matched several lines, it restarted scanning at the |
| 930 | next line instead of moving on to the end of the matched string, which can be |
| 931 | several lines after the start. |
| 932 | |
| 933 | 56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line |
| 934 | with updates to the non-Windows version. |
| 935 | |
| 936 | |
| 937 | |
Janis Danisevskis | 8b979b2 | 2016-08-15 16:09:16 +0100 | [diff] [blame] | 938 | Version 10.22 29-July-2016 |
| 939 | -------------------------- |
| 940 | |
| 941 | 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3 |
| 942 | to fix problems with running the tests under Windows. |
| 943 | |
| 944 | 2. Implemented a facility for quoting literal characters within hexadecimal |
| 945 | patterns in pcre2test, to make it easier to create patterns with just a few |
| 946 | non-printing characters. |
| 947 | |
| 948 | 3. Binary zeros are not supported in pcre2test input files. It now detects them |
| 949 | and gives an error. |
| 950 | |
| 951 | 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to |
| 952 | smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so |
| 953 | that it matches only unknown objects. |
| 954 | |
| 955 | 5. Updated the maintenance script maint/ManyConfigTests to make it easier to |
| 956 | select individual groups of tests. |
| 957 | |
| 958 | 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option |
| 959 | used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this |
| 960 | disables the use of back references (and subroutine calls), which are supported |
| 961 | by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no |
| 962 | longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch |
| 963 | and pmatch when regexec() is called. |
| 964 | |
| 965 | 7. Because of 6 above, pcre2test has been modified with a new modifier called |
| 966 | posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture |
| 967 | modifier had this effect. That option is now ignored when the POSIX API is in |
| 968 | use. |
| 969 | |
| 970 | 8. Minor tidies to the pcre2demo.c sample program, including more comments |
| 971 | about its 8-bit-ness. |
| 972 | |
| 973 | 9. Detect unmatched closing parentheses and give the error in the pre-scan |
| 974 | instead of later. Previously the pre-scan carried on and could give a |
| 975 | misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a |
| 976 | message about invalid duplicate group names. |
| 977 | |
| 978 | 10. It has happened that pcre2test was accidentally linked with another POSIX |
| 979 | regex library instead of libpcre2-posix. In this situation, a call to regcomp() |
| 980 | (in the other library) may succeed, returning zero, but of course putting its |
| 981 | own data into the regex_t block. In one example the re_pcre2_code field was |
| 982 | left as NULL, which made pcre2test think it had not got a compiled POSIX regex, |
| 983 | so it treated the next line as another pattern line, resulting in a confusing |
| 984 | error message. A check has been added to pcre2test to see if the data returned |
| 985 | from a successful call of regcomp() are valid for PCRE2's regcomp(). If they |
| 986 | are not, an error message is output and the pcre2test run is abandoned. The |
| 987 | message points out the possibility of a mis-linking. Hopefully this will avoid |
| 988 | some head-scratching the next time this happens. |
| 989 | |
| 990 | 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind |
| 991 | assertion, caused pcre2test to output a very large number of spaces when the |
| 992 | callout was taken, making the program appearing to loop. |
| 993 | |
| 994 | 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply |
| 995 | nested set of parentheses of sufficient size caused an overflow of the |
| 996 | compiling workspace (which was diagnosed, but of course is not desirable). |
| 997 | |
| 998 | 13. Detect missing closing parentheses during the pre-pass for group |
| 999 | identification. |
| 1000 | |
| 1001 | 14. Changed some integer variable types and put in a number of casts, following |
| 1002 | a report of compiler warnings from Visual Studio 2013 and a few tests with |
| 1003 | gcc's -Wconversion (which still throws up a lot). |
| 1004 | |
| 1005 | 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test |
| 1006 | for testing it. |
| 1007 | |
| 1008 | 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of |
| 1009 | regerror(). When the error buffer is too small, my version of snprintf() puts a |
| 1010 | binary zero in the final byte. Bug #1801 seems to show that other versions do |
| 1011 | not do this, leading to bad output from pcre2test when it was checking for |
| 1012 | buffer overflow. It no longer assumes a binary zero at the end of a too-small |
| 1013 | regerror() buffer. |
| 1014 | |
| 1015 | 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not |
| 1016 | actually affect anything, by sheer luck. |
| 1017 | |
| 1018 | 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect |
| 1019 | "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for |
| 1020 | older MSVC compilers. This has been done both in src/pcre2_internal.h for most |
| 1021 | of the library, and also in src/pcre2posix.c, which no longer includes |
| 1022 | pcre2_internal.h (see 24 below). |
| 1023 | |
| 1024 | 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC |
| 1025 | static compilation. Subsequently applied Chris Wilson's second patch, putting |
| 1026 | the first patch under a new option instead of being unconditional when |
| 1027 | PCRE_STATIC is set. |
| 1028 | |
| 1029 | 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not |
| 1030 | to convert \r\n at the ends of reflected lines into \r\r\n. This required |
| 1031 | ensuring that other output that is written to stdout (e.g. file names) uses the |
| 1032 | appropriate line terminator: \r\n for Windows, \n otherwise. |
| 1033 | |
| 1034 | 21. When a line is too long for pcre2grep's internal buffer, show the maximum |
| 1035 | length in the error message. |
| 1036 | |
| 1037 | 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH |
| 1038 | additions). |
| 1039 | |
| 1040 | 23. RunTest.bat was missing a "set type" line for test 22. |
| 1041 | |
| 1042 | 24. The pcre2posix.c file was including pcre2_internal.h, and using some |
| 1043 | "private" knowledge of the data structures. This is unnecessary; the code has |
| 1044 | been re-factored and no longer includes pcre2_internal.h. |
| 1045 | |
| 1046 | 25. A racing condition is fixed in JIT reported by Mozilla. |
| 1047 | |
| 1048 | 26. Minor code refactor to avoid "array subscript is below array bounds" |
| 1049 | compiler warning. |
| 1050 | |
| 1051 | 27. Minor code refactor to avoid "left shift of negative number" warning. |
| 1052 | |
| 1053 | 28. Add a bit more sanity checking to pcre2_serialize_decode() and document |
| 1054 | that it expects trusted data. |
| 1055 | |
| 1056 | 29. Fix typo in pcre2_jit_test.c |
| 1057 | |
| 1058 | 30. Due to an oversight, pcre2grep was not making use of JIT when available. |
| 1059 | This is now fixed. |
| 1060 | |
| 1061 | 31. The RunGrepTest script is updated to use the valgrind suppressions file |
| 1062 | when testing with JIT under valgrind (compare 10.21/51 below). The suppressions |
| 1063 | file is updated so that is now the same as for PCRE1: it suppresses the |
| 1064 | Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled |
| 1065 | code). Also changed smc-check=all to smc-check=all-non-file as was done for |
| 1066 | RunTest (see 4 above). |
| 1067 | |
| 1068 | 32. Implemented the PCRE2_NO_JIT option for pcre2_match(). |
| 1069 | |
| 1070 | 33. Fix typo that gave a compiler error when JIT not supported. |
| 1071 | |
| 1072 | 34. Fix comment describing the returns from find_fixedlength(). |
| 1073 | |
| 1074 | 35. Fix potential negative index in pcre2test. |
| 1075 | |
| 1076 | 36. Calls to pcre2_get_error_message() with error numbers that are never |
| 1077 | returned by PCRE2 functions were returning empty strings. Now the error code |
| 1078 | PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to |
| 1079 | show the texts for given error numbers (i.e. to call pcre2_get_error_message() |
| 1080 | and display what it returns) and a few representative error codes are now |
| 1081 | checked in RunTest. |
| 1082 | |
| 1083 | 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in |
| 1084 | pcre2_match.c, in anticipation that this is needed for the same reason it was |
| 1085 | recently added to pcrecpp.cc in PCRE1. |
| 1086 | |
| 1087 | 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when |
| 1088 | the match extended over a line boundary, as it tried to find more matches "on |
| 1089 | the same line" - but it was already over the end. |
| 1090 | |
| 1091 | 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it |
| 1092 | to the same code as '.' when PCRE2_DOTALL is set). |
| 1093 | |
| 1094 | 40. Fix two clang compiler warnings in pcre2test when only one code unit width |
| 1095 | is supported. |
| 1096 | |
Elliott Hughes | 653c210 | 2019-01-09 15:41:36 -0800 | [diff] [blame] | 1097 | 41. Upgrade RunTest to automatically re-run test 2 with a large (64MiB) stack |
| 1098 | if it fails when running the interpreter with a 16MiB stack (and if changing |
| 1099 | the stack size via pcre2test is possible). This avoids having to manually set a |
Janis Danisevskis | 8b979b2 | 2016-08-15 16:09:16 +0100 | [diff] [blame] | 1100 | large stack size when testing with clang. |
| 1101 | |
| 1102 | 42. Fix register overwite in JIT when SSE2 acceleration is enabled. |
| 1103 | |
| 1104 | 43. Detect integer overflow in pcre2test pattern and data repetition counts. |
| 1105 | |
| 1106 | 44. In pcre2test, ignore "allcaptures" after DFA matching. |
| 1107 | |
| 1108 | 45. Fix unaligned accesses on x86. Patch by Marc Mutz. |
| 1109 | |
| 1110 | 46. Fix some more clang compiler warnings. |
| 1111 | |
| 1112 | |
Janis Danisevskis | 53e448c | 2016-03-31 13:35:25 +0100 | [diff] [blame] | 1113 | Version 10.21 12-January-2016 |
| 1114 | ----------------------------- |
| 1115 | |
| 1116 | 1. Improve matching speed of patterns starting with + or * in JIT. |
| 1117 | |
| 1118 | 2. Use memchr() to find the first character in an unanchored match in 8-bit |
| 1119 | mode in the interpreter. This gives a significant speed improvement. |
| 1120 | |
| 1121 | 3. Removed a redundant copy of the opcode_possessify table in the |
| 1122 | pcre2_auto_possessify.c source. |
| 1123 | |
| 1124 | 4. Fix typos in dftables.c for z/OS. |
| 1125 | |
| 1126 | 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that |
| 1127 | processing them could involve a buffer overflow if the following character was |
| 1128 | an opening parenthesis. |
| 1129 | |
| 1130 | 6. Change 36 for 10.20 also introduced a bug in processing this pattern: |
| 1131 | /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK) |
| 1132 | setting (which (*:0) is), then (?x) did not get unset at the end of its group |
| 1133 | during the scan for named groups, and hence the external # was incorrectly |
| 1134 | treated as a comment and the invalid (?' at the end of the pattern was not |
| 1135 | diagnosed. This caused a buffer overflow during the real compile. This bug was |
| 1136 | discovered by Karl Skomski with the LLVM fuzzer. |
| 1137 | |
| 1138 | 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its |
| 1139 | own source module to avoid a circular dependency between src/pcre2_compile.c |
| 1140 | and src/pcre2_study.c |
| 1141 | |
| 1142 | 8. A callout with a string argument containing an opening square bracket, for |
| 1143 | example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer |
| 1144 | overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer. |
| 1145 | |
| 1146 | 9. The handling of callouts during the pre-pass for named group identification |
| 1147 | has been tightened up. |
| 1148 | |
| 1149 | 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or |
| 1150 | possessive. This is a very minor optimization. |
| 1151 | |
| 1152 | 11. A possessively repeated conditional group that could match an empty string, |
| 1153 | for example, /(?(R))*+/, was incorrectly compiled. |
| 1154 | |
| 1155 | 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian |
| 1156 | Persch). |
| 1157 | |
| 1158 | 13. An empty comment (?#) in a pattern was incorrectly processed and could |
| 1159 | provoke a buffer overflow. This bug was discovered by Karl Skomski with the |
| 1160 | LLVM fuzzer. |
| 1161 | |
| 1162 | 14. Fix infinite recursion in the JIT compiler when certain patterns such as |
| 1163 | /(?:|a|){100}x/ are analysed. |
| 1164 | |
| 1165 | 15. Some patterns with character classes involving [: and \\ were incorrectly |
| 1166 | compiled and could cause reading from uninitialized memory or an incorrect |
| 1167 | error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The |
| 1168 | first of these bugs was discovered by Karl Skomski with the LLVM fuzzer. |
| 1169 | |
| 1170 | 16. Pathological patterns containing many nested occurrences of [: caused |
| 1171 | pcre2_compile() to run for a very long time. This bug was found by the LLVM |
| 1172 | fuzzer. |
| 1173 | |
| 1174 | 17. A missing closing parenthesis for a callout with a string argument was not |
| 1175 | being diagnosed, possibly leading to a buffer overflow. This bug was found by |
| 1176 | the LLVM fuzzer. |
| 1177 | |
| 1178 | 18. A conditional group with only one branch has an implicit empty alternative |
| 1179 | branch and must therefore be treated as potentially matching an empty string. |
| 1180 | |
| 1181 | 19. If (?R was followed by - or + incorrect behaviour happened instead of a |
| 1182 | diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer. |
| 1183 | |
| 1184 | 20. Another bug that was introduced by change 36 for 10.20: conditional groups |
| 1185 | whose condition was an assertion preceded by an explicit callout with a string |
| 1186 | argument might be incorrectly processed, especially if the string contained \Q. |
| 1187 | This bug was discovered by Karl Skomski with the LLVM fuzzer. |
| 1188 | |
| 1189 | 21. Compiling PCRE2 with the sanitize options of clang showed up a number of |
| 1190 | very pedantic coding infelicities and a buffer overflow while checking a UTF-8 |
| 1191 | string if the final multi-byte UTF-8 character was truncated. |
| 1192 | |
| 1193 | 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a |
| 1194 | class, where both values are literal letters in the same case, omit the |
| 1195 | non-letter EBCDIC code points within the range. |
| 1196 | |
| 1197 | 23. Finding the minimum matching length of complex patterns with back |
| 1198 | references and/or recursions can take a long time. There is now a cut-off that |
| 1199 | gives up trying to find a minimum length when things get too complex. |
| 1200 | |
| 1201 | 24. An optimization has been added that speeds up finding the minimum matching |
| 1202 | length for patterns containing repeated capturing groups or recursions. |
| 1203 | |
| 1204 | 25. If a pattern contained a back reference to a group whose number was |
| 1205 | duplicated as a result of appearing in a (?|...) group, the computation of the |
| 1206 | minimum matching length gave a wrong result, which could cause incorrect "no |
| 1207 | match" errors. For such patterns, a minimum matching length cannot at present |
| 1208 | be computed. |
| 1209 | |
| 1210 | 26. Added a check for integer overflow in conditions (?(<digits>) and |
| 1211 | (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM |
| 1212 | fuzzer. |
| 1213 | |
| 1214 | 27. Fixed an issue when \p{Any} inside an xclass did not read the current |
| 1215 | character. |
| 1216 | |
| 1217 | 28. If pcre2grep was given the -q option with -c or -l, or when handling a |
| 1218 | binary file, it incorrectly wrote output to stdout. |
| 1219 | |
| 1220 | 29. The JIT compiler did not restore the control verb head in case of *THEN |
| 1221 | control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer. |
| 1222 | |
| 1223 | 30. The way recursive references such as (?3) are compiled has been re-written |
| 1224 | because the old way was the cause of many issues. Now, conversion of the group |
| 1225 | number into a pattern offset does not happen until the pattern has been |
| 1226 | completely compiled. This does mean that detection of all infinitely looping |
| 1227 | recursions is postponed till match time. In the past, some easy ones were |
| 1228 | detected at compile time. This re-writing was done in response to yet another |
| 1229 | bug found by the LLVM fuzzer. |
| 1230 | |
| 1231 | 31. A test for a back reference to a non-existent group was missing for items |
| 1232 | such as \987. This caused incorrect code to be compiled. This issue was found |
| 1233 | by Karl Skomski with a custom LLVM fuzzer. |
| 1234 | |
| 1235 | 32. Error messages for syntax errors following \g and \k were giving inaccurate |
| 1236 | offsets in the pattern. |
| 1237 | |
| 1238 | 33. Improve the performance of starting single character repetitions in JIT. |
| 1239 | |
| 1240 | 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0. |
| 1241 | |
| 1242 | 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now |
| 1243 | give the right offset instead of zero. |
| 1244 | |
| 1245 | 36. The JIT compiler should not check repeats after a {0,1} repeat byte code. |
| 1246 | This issue was found by Karl Skomski with a custom LLVM fuzzer. |
| 1247 | |
| 1248 | 37. The JIT compiler should restore the control chain for empty possessive |
| 1249 | repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer. |
| 1250 | |
| 1251 | 38. A bug which was introduced by the single character repetition optimization |
| 1252 | was fixed. |
| 1253 | |
| 1254 | 39. Match limit check added to recursion. This issue was found by Karl Skomski |
| 1255 | with a custom LLVM fuzzer. |
| 1256 | |
| 1257 | 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look |
| 1258 | only at the part of the subject that is relevant when the starting offset is |
| 1259 | non-zero. |
| 1260 | |
| 1261 | 41. Improve first character match in JIT with SSE2 on x86. |
| 1262 | |
| 1263 | 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski |
| 1264 | with a custom LLVM fuzzer. |
| 1265 | |
| 1266 | 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy |
| 1267 | III). |
| 1268 | |
| 1269 | 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added |
| 1270 | test (there are now 20 in total). |
| 1271 | |
| 1272 | 45. Fixed a corner case of range optimization in JIT. |
| 1273 | |
| 1274 | 46. Add the ${*MARK} facility to pcre2_substitute(). |
| 1275 | |
| 1276 | 47. Modifier lists in pcre2test were splitting at spaces without the required |
| 1277 | commas. |
| 1278 | |
| 1279 | 48. Implemented PCRE2_ALT_VERBNAMES. |
| 1280 | |
| 1281 | 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom |
| 1282 | LLVM fuzzer. |
| 1283 | |
| 1284 | 50. The pcre2test program has been extended by adding the #newline_default |
| 1285 | command. This has made it possible to run the standard tests when PCRE2 is |
| 1286 | compiled with either CR or CRLF as the default newline convention. As part of |
| 1287 | this work, the new command was added to several test files and the testing |
| 1288 | scripts were modified. The pcre2grep tests can now also be run when there is no |
| 1289 | LF in the default newline convention. |
| 1290 | |
| 1291 | 51. The RunTest script has been modified so that, when JIT is used and valgrind |
| 1292 | is specified, a valgrind suppressions file is set up to ignore "Invalid read of |
| 1293 | size 16" errors because these are false positives when the hardware supports |
| 1294 | the SSE2 instruction set. |
| 1295 | |
| 1296 | 52. It is now possible to have comment lines amid the subject strings in |
| 1297 | pcre2test (and perltest.sh) input. |
| 1298 | |
| 1299 | 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit(). |
| 1300 | |
| 1301 | 54. Add the null_context modifier to pcre2test so that calling pcre2_compile() |
| 1302 | and the matching functions with NULL contexts can be tested. |
| 1303 | |
| 1304 | 55. Implemented PCRE2_SUBSTITUTE_EXTENDED. |
| 1305 | |
| 1306 | 56. In a character class such as [\W\p{Any}] where both a negative-type escape |
| 1307 | ("not a word character") and a property escape were present, the property |
| 1308 | escape was being ignored. |
| 1309 | |
| 1310 | 57. Fixed integer overflow for patterns whose minimum matching length is very, |
| 1311 | very large. |
| 1312 | |
| 1313 | 58. Implemented --never-backslash-C. |
| 1314 | |
| 1315 | 59. Change 55 above introduced a bug by which certain patterns provoked the |
| 1316 | erroneous error "\ at end of pattern". |
| 1317 | |
| 1318 | 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling |
| 1319 | errors or other strange effects if compiled in UCP mode. Found with libFuzzer |
| 1320 | and AddressSanitizer. |
| 1321 | |
| 1322 | 61. Whitespace at the end of a pcre2test pattern line caused a spurious error |
| 1323 | message if there were only single-character modifiers. It should be ignored. |
| 1324 | |
| 1325 | 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results |
| 1326 | or segmentation errors for some patterns. Found with libFuzzer and |
| 1327 | AddressSanitizer. |
| 1328 | |
| 1329 | 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer |
| 1330 | overflow. |
| 1331 | |
| 1332 | 64. Improve error message for overly-complicated patterns. |
| 1333 | |
| 1334 | 65. Implemented an optional replication feature for patterns in pcre2test, to |
| 1335 | make it easier to test long repetitive patterns. The tests for 63 above are |
| 1336 | converted to use the new feature. |
| 1337 | |
| 1338 | 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could |
| 1339 | misbehave. |
| 1340 | |
| 1341 | 67. In pcre2_substitute() in UTF mode, the UTF validity check on the |
| 1342 | replacement string was happening before the length setting when the replacement |
| 1343 | string was zero-terminated. |
| 1344 | |
| 1345 | 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the |
| 1346 | second and subsequent calls to pcre2_match(). |
| 1347 | |
| 1348 | 69. There was no check for integer overflow for a replacement group number in |
| 1349 | pcre2_substitute(). An added check for a number greater than the largest group |
| 1350 | number in the pattern means this is not now needed. |
| 1351 | |
| 1352 | 70. The PCRE2-specific VERSION condition didn't work correctly if only one |
| 1353 | digit was given after the decimal point, or if more than two digits were given. |
| 1354 | It now works with one or two digits, and gives a compile time error if more are |
| 1355 | given. |
| 1356 | |
| 1357 | 71. In pcre2_substitute() there was the possibility of reading one code unit |
| 1358 | beyond the end of the replacement string. |
| 1359 | |
| 1360 | 72. The code for checking a subject's UTF-32 validity for a pattern with a |
| 1361 | lookbehind involved an out-of-bounds pointer, which could potentially cause |
| 1362 | trouble in some environments. |
| 1363 | |
| 1364 | 73. The maximum lookbehind length was incorrectly calculated for patterns such |
| 1365 | as /(?<=(a)(?-1))x/ which have a recursion within a backreference. |
| 1366 | |
| 1367 | 74. Give an error if a lookbehind assertion is longer than 65535 code units. |
| 1368 | |
| 1369 | 75. Give an error in pcre2_substitute() if a match ends before it starts (as a |
| 1370 | result of the use of \K). |
| 1371 | |
| 1372 | 76. Check the length of subpattern names and the names in (*MARK:xx) etc. |
| 1373 | dynamically to avoid the possibility of integer overflow. |
| 1374 | |
| 1375 | 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the |
| 1376 | size of patterns that they are prepared to handle. |
| 1377 | |
| 1378 | 78. (*NO_AUTO_POSSESS) was not working. |
| 1379 | |
| 1380 | 79. Adding group information caching improves the speed of compiling when |
| 1381 | checking whether a group has a fixed length and/or could match an empty string, |
| 1382 | especially when recursion or subroutine calls are involved. However, this |
| 1383 | cannot be used when (?| is present in the pattern because the same number may |
| 1384 | be used for groups of different sizes. To catch runaway patterns in this |
| 1385 | situation, counts have been introduced to the functions that scan for empty |
| 1386 | branches or compute fixed lengths. |
| 1387 | |
| 1388 | 80. Allow for the possibility of the size of the nest_save structure not being |
| 1389 | a factor of the size of the compiling workspace (it currently is). |
| 1390 | |
| 1391 | 81. Check for integer overflow in minimum length calculation and cap it at |
| 1392 | 65535. |
| 1393 | |
| 1394 | 82. Small optimizations in code for finding the minimum matching length. |
| 1395 | |
| 1396 | 83. Lock out configuring for EBCDIC with non-8-bit libraries. |
| 1397 | |
| 1398 | 84. Test for error code <= 0 in regerror(). |
| 1399 | |
| 1400 | 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute(). |
| 1401 | |
| 1402 | 86. Avoid the possibility of computing with an out-of-bounds pointer (though |
| 1403 | not dereferencing it) while handling lookbehind assertions. |
| 1404 | |
| 1405 | 87. Failure to get memory for the match data in regcomp() is now given as a |
| 1406 | regcomp() error instead of waiting for regexec() to pick it up. |
| 1407 | |
| 1408 | 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid |
| 1409 | newline sequence. |
| 1410 | |
| 1411 | 89. Paranoid check in regcomp() for bad error code from pcre2_compile(). |
| 1412 | |
| 1413 | 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well |
| 1414 | as for link size 2. |
| 1415 | |
| 1416 | 91. Document that JIT has a limit on pattern size, and give more information |
| 1417 | about JIT compile failures in pcre2test. |
| 1418 | |
| 1419 | 92. Implement PCRE2_INFO_HASBACKSLASHC. |
| 1420 | |
| 1421 | 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports |
| 1422 | with JIT (possibly caused by SSE2?). |
| 1423 | |
| 1424 | 94. Support offset_limit in JIT. |
| 1425 | |
| 1426 | 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed |
| 1427 | by a single ASCII character in a class item, was incorrectly compiled in UCP |
| 1428 | mode. The POSIX class got lost, but only if the single character followed it. |
| 1429 | |
| 1430 | 96. [:punct:] in UCP mode was matching some characters in the range 128-255 |
| 1431 | that should not have been matched. |
| 1432 | |
| 1433 | 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all |
| 1434 | characters with code points greater than 255 are in the class. When a Unicode |
| 1435 | property was also in the class (if PCRE2_UCP is set, escapes such as \w are |
| 1436 | turned into Unicode properties), wide characters were not correctly handled, |
| 1437 | and could fail to match. |
| 1438 | |
| 1439 | 98. In pcre2test, make the "startoffset" modifier a synonym of "offset", |
| 1440 | because it sets the "startoffset" parameter for pcre2_match(). |
| 1441 | |
| 1442 | 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between |
| 1443 | an item and its qualifier (for example, A(?#comment)?B) pcre2_compile() |
| 1444 | misbehaved. This bug was found by the LLVM fuzzer. |
| 1445 | |
| 1446 | 100. The error for an invalid UTF pattern string always gave the code unit |
| 1447 | offset as zero instead of where the invalidity was found. |
| 1448 | |
| 1449 | 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not |
| 1450 | working correctly in UCP mode. |
| 1451 | |
| 1452 | 102. Similar to 99 above, if an isolated \E was present between an item and its |
| 1453 | qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug |
| 1454 | was found by the LLVM fuzzer. |
| 1455 | |
| 1456 | 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND |
| 1457 | was set when the pmatch argument was NULL. It now returns REG_INVARG. |
| 1458 | |
| 1459 | 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep. |
| 1460 | |
| 1461 | 105. An empty \Q\E sequence between an item and its qualifier caused |
| 1462 | pcre2_compile() to misbehave when auto callouts were enabled. This bug |
| 1463 | was found by the LLVM fuzzer. |
| 1464 | |
| 1465 | 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or |
| 1466 | other verb "name" ended with whitespace immediately before the closing |
| 1467 | parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when |
| 1468 | both those options were set. |
| 1469 | |
| 1470 | 107. In a number of places pcre2_compile() was not handling NULL characters |
| 1471 | correctly, and pcre2test with the "bincode" modifier was not always correctly |
| 1472 | displaying fields containing NULLS: |
| 1473 | |
| 1474 | (a) Within /x extended #-comments |
| 1475 | (b) Within the "name" part of (*MARK) and other *verbs |
| 1476 | (c) Within the text argument of a callout |
| 1477 | |
| 1478 | 108. If a pattern that was compiled with PCRE2_EXTENDED started with white |
| 1479 | space or a #-type comment that was followed by (?-x), which turns off |
| 1480 | PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again, |
| 1481 | pcre2_compile() assumed that (?-x) applied to the whole pattern and |
| 1482 | consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix |
Janis Danisevskis | 8b979b2 | 2016-08-15 16:09:16 +0100 | [diff] [blame] | 1483 | for this bug means that a setting of any of the (?imsxJU) options at the start |
Janis Danisevskis | 53e448c | 2016-03-31 13:35:25 +0100 | [diff] [blame] | 1484 | of a pattern is no longer transferred to the options that are returned by |
| 1485 | PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have |
| 1486 | changed when the effects of those options were all moved to compile time. |
| 1487 | |
| 1488 | 109. An escaped closing parenthesis in the "name" part of a (*verb) when |
| 1489 | PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug |
| 1490 | was found by the LLVM fuzzer. |
| 1491 | |
| 1492 | 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it |
| 1493 | possible to test it. |
| 1494 | |
| 1495 | 111. "Harden" pcre2test against ridiculously large values in modifiers and |
| 1496 | command line arguments. |
| 1497 | |
| 1498 | 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_ |
| 1499 | LENGTH. |
| 1500 | |
| 1501 | 113. Fix printing of *MARK names that contain binary zeroes in pcre2test. |
| 1502 | |
| 1503 | |
| 1504 | Version 10.20 30-June-2015 |
| 1505 | -------------------------- |
| 1506 | |
| 1507 | 1. Callouts with string arguments have been added. |
| 1508 | |
| 1509 | 2. Assertion code generator in JIT has been optimized. |
| 1510 | |
| 1511 | 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The |
| 1512 | pcre2_compile() function read past the end of the input before diagnosing an |
| 1513 | error. This bug was discovered by the LLVM fuzzer. |
| 1514 | |
| 1515 | 4. Implemented pcre2_callout_enumerate(). |
| 1516 | |
| 1517 | 5. Fix JIT compilation of conditional blocks whose assertion is converted to |
| 1518 | (*FAIL). E.g: /(?(?!))/. |
| 1519 | |
| 1520 | 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was |
| 1521 | discovered by the LLVM fuzzer. |
| 1522 | |
| 1523 | 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly |
| 1524 | when this assertion was used as a condition, for example (?(?!)a|b). In |
| 1525 | pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect |
| 1526 | error about an unsupported item. |
| 1527 | |
| 1528 | 8. For some types of pattern, for example /Z*(|d*){216}/, the auto- |
| 1529 | possessification code could take exponential time to complete. A recursion |
| 1530 | depth limit of 1000 has been imposed to limit the resources used by this |
| 1531 | optimization. This infelicity was discovered by the LLVM fuzzer. |
| 1532 | |
| 1533 | 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class |
| 1534 | such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored |
| 1535 | because \S ensures they are all in the class. The code for doing this was |
| 1536 | interacting badly with the code for computing the amount of space needed to |
| 1537 | compile the pattern, leading to a buffer overflow. This bug was discovered by |
| 1538 | the LLVM fuzzer. |
| 1539 | |
| 1540 | 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside |
| 1541 | other kinds of group caused stack overflow at compile time. This bug was |
| 1542 | discovered by the LLVM fuzzer. |
| 1543 | |
| 1544 | 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment |
| 1545 | between a subroutine call and its quantifier was incorrectly compiled, leading |
| 1546 | to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer. |
| 1547 | |
| 1548 | 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an |
| 1549 | assertion after (?(. The code was failing to check the character after (?(?< |
| 1550 | for the ! or = that would indicate a lookbehind assertion. This bug was |
| 1551 | discovered by the LLVM fuzzer. |
| 1552 | |
| 1553 | 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with |
| 1554 | a fixed maximum following a group that contains a subroutine reference was |
| 1555 | incorrectly compiled and could trigger buffer overflow. This bug was discovered |
| 1556 | by the LLVM fuzzer. |
| 1557 | |
| 1558 | 14. Negative relative recursive references such as (?-7) to non-existent |
| 1559 | subpatterns were not being diagnosed and could lead to unpredictable behaviour. |
| 1560 | This bug was discovered by the LLVM fuzzer. |
| 1561 | |
| 1562 | 15. The bug fixed in 14 was due to an integer variable that was unsigned when |
| 1563 | it should have been signed. Some other "int" variables, having been checked, |
| 1564 | have either been changed to uint32_t or commented as "must be signed". |
| 1565 | |
| 1566 | 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1))) |
| 1567 | caused a stack overflow instead of the diagnosis of a non-fixed length |
| 1568 | lookbehind assertion. This bug was discovered by the LLVM fuzzer. |
| 1569 | |
| 1570 | 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern |
| 1571 | (e.g. /(?<=\Ka)/) could make pcre2grep loop. |
| 1572 | |
| 1573 | 18. There was a similar problem to 17 in pcre2test for global matches, though |
| 1574 | the code there did catch the loop. |
| 1575 | |
| 1576 | 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*), |
| 1577 | and a subsequent item in the pattern caused a non-match, backtracking over the |
| 1578 | repeated \X did not stop, but carried on past the start of the subject, causing |
| 1579 | reference to random memory and/or a segfault. There were also some other cases |
| 1580 | where backtracking after \C could crash. This set of bugs was discovered by the |
| 1581 | LLVM fuzzer. |
| 1582 | |
| 1583 | 20. The function for finding the minimum length of a matching string could take |
| 1584 | a very long time if mutual recursion was present many times in a pattern, for |
| 1585 | example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has |
| 1586 | been implemented. This infelicity was discovered by the LLVM fuzzer. |
| 1587 | |
| 1588 | 21. Implemented PCRE2_NEVER_BACKSLASH_C. |
| 1589 | |
| 1590 | 22. The feature for string replication in pcre2test could read from freed |
| 1591 | memory if the replication required a buffer to be extended, and it was not |
| 1592 | working properly in 16-bit and 32-bit modes. This issue was discovered by a |
| 1593 | fuzzer: see http://lcamtuf.coredump.cx/afl/. |
| 1594 | |
| 1595 | 23. Added the PCRE2_ALT_CIRCUMFLEX option. |
| 1596 | |
| 1597 | 24. Adjust the treatment of \8 and \9 to be the same as the current Perl |
| 1598 | behaviour. |
| 1599 | |
| 1600 | 25. Static linking against the PCRE2 library using the pkg-config module was |
| 1601 | failing on missing pthread symbols. |
| 1602 | |
| 1603 | 26. If a group that contained a recursive back reference also contained a |
| 1604 | forward reference subroutine call followed by a non-forward-reference |
| 1605 | subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to |
| 1606 | compile correct code, leading to undefined behaviour or an internally detected |
| 1607 | error. This bug was discovered by the LLVM fuzzer. |
| 1608 | |
| 1609 | 27. Quantification of certain items (e.g. atomic back references) could cause |
| 1610 | incorrect code to be compiled when recursive forward references were involved. |
| 1611 | For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was |
| 1612 | discovered by the LLVM fuzzer. |
| 1613 | |
| 1614 | 28. A repeated conditional group whose condition was a reference by name caused |
| 1615 | a buffer overflow if there was more than one group with the given name. This |
| 1616 | bug was discovered by the LLVM fuzzer. |
| 1617 | |
| 1618 | 29. A recursive back reference by name within a group that had the same name as |
| 1619 | another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/. |
| 1620 | This bug was discovered by the LLVM fuzzer. |
| 1621 | |
| 1622 | 30. A forward reference by name to a group whose number is the same as the |
| 1623 | current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a |
| 1624 | buffer overflow at compile time. This bug was discovered by the LLVM fuzzer. |
| 1625 | |
| 1626 | 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1 |
| 1627 | as an int; fixed by writing it as 1u). |
| 1628 | |
| 1629 | 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives |
| 1630 | a warning for "fileno" unless -std=gnu99 us used. |
| 1631 | |
| 1632 | 33. A lookbehind assertion within a set of mutually recursive subpatterns could |
| 1633 | provoke a buffer overflow. This bug was discovered by the LLVM fuzzer. |
| 1634 | |
| 1635 | 34. Give an error for an empty subpattern name such as (?''). |
| 1636 | |
| 1637 | 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains |
| 1638 | \P, \p, or \X. |
| 1639 | |
| 1640 | 36. The way named subpatterns are handled has been refactored. There is now a |
| 1641 | pre-pass over the regex which does nothing other than identify named |
| 1642 | subpatterns and count the total captures. This means that information about |
| 1643 | named patterns is known before the rest of the compile. In particular, it means |
| 1644 | that forward references can be checked as they are encountered. Previously, the |
| 1645 | code for handling forward references was contorted and led to several errors in |
| 1646 | computing the memory requirements for some patterns, leading to buffer |
| 1647 | overflows. |
| 1648 | |
| 1649 | 37. There was no check for integer overflow in subroutine calls such as (?123). |
| 1650 | |
| 1651 | 38. The table entry for \l in EBCDIC environments was incorrect, leading to its |
| 1652 | being treated as a literal 'l' instead of causing an error. |
| 1653 | |
| 1654 | 39. If a non-capturing group containing a conditional group that could match |
| 1655 | an empty string was repeated, it was not identified as matching an empty string |
| 1656 | itself. For example: /^(?:(?(1)x|)+)+$()/. |
| 1657 | |
| 1658 | 40. In an EBCDIC environment, pcretest was mishandling the escape sequences |
| 1659 | \a and \e in test subject lines. |
| 1660 | |
| 1661 | 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII |
| 1662 | instead of the EBCDIC value. |
| 1663 | |
| 1664 | 42. The handling of \c in an EBCDIC environment has been revised so that it is |
| 1665 | now compatible with the specification in Perl's perlebcdic page. |
| 1666 | |
| 1667 | 43. Single character repetition in JIT has been improved. 20-30% speedup |
| 1668 | was achieved on certain patterns. |
| 1669 | |
| 1670 | 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in |
| 1671 | ASCII/Unicode. This has now been added to the list of characters that are |
| 1672 | recognized as white space in EBCDIC. |
| 1673 | |
| 1674 | 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave |
| 1675 | an error (correctly) when used outside a class, but did not give an error |
| 1676 | within a class. |
| 1677 | |
| 1678 | 46. \h within a class was incorrectly compiled in EBCDIC environments. |
| 1679 | |
| 1680 | 47. JIT should return with error when the compiled pattern requires |
| 1681 | more stack space than the maximum. |
| 1682 | |
| 1683 | 48. Fixed a memory leak in pcre2grep when a locale is set. |
| 1684 | |
| 1685 | |
| 1686 | Version 10.10 06-March-2015 |
| 1687 | --------------------------- |
| 1688 | |
| 1689 | 1. When a pattern is compiled, it remembers the highest back reference so that |
| 1690 | when matching, if the ovector is too small, extra memory can be obtained to |
| 1691 | use instead. A conditional subpattern whose condition is a check on a capture |
| 1692 | having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is |
| 1693 | another kind of back reference, but it was not setting the highest |
| 1694 | backreference number. This mattered only if pcre2_match() was called with an |
| 1695 | ovector that was too small to hold the capture, and there was no other kind of |
| 1696 | back reference (a situation which is probably quite rare). The effect of the |
| 1697 | bug was that the condition was always treated as FALSE when the capture could |
| 1698 | not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug |
| 1699 | has been fixed. |
| 1700 | |
| 1701 | 2. Functions for serialization and deserialization of sets of compiled patterns |
| 1702 | have been added. |
| 1703 | |
| 1704 | 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove |
| 1705 | excess code units at the end of the data block that may occasionally occur if |
| 1706 | the code for calculating the size over-estimates. This change stops the |
| 1707 | serialization code copying uninitialized data, to which valgrind objects. The |
| 1708 | documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not |
| 1709 | include the general overhead. This has been corrected. |
| 1710 | |
| 1711 | 4. All code units in every slot in the table of group names are now set, again |
| 1712 | in order to avoid accessing uninitialized data when serializing. |
| 1713 | |
| 1714 | 5. The (*NO_JIT) feature is implemented. |
| 1715 | |
| 1716 | 6. If a bug that caused pcre2_compile() to use more memory than allocated was |
| 1717 | triggered when using valgrind, the code in (3) above passed a stupidly large |
| 1718 | value to valgrind. This caused a crash instead of an "internal error" return. |
| 1719 | |
| 1720 | 7. A reference to a duplicated named group (either a back reference or a test |
| 1721 | for being set in a conditional) that occurred in a part of the pattern where |
| 1722 | PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern |
| 1723 | to be incorrectly calculated, leading to overwriting. |
| 1724 | |
| 1725 | 8. A mutually recursive set of back references such as (\2)(\1) caused a |
| 1726 | segfault at compile time (while trying to find the minimum matching length). |
| 1727 | The infinite loop is now broken (with the minimum length unset, that is, zero). |
| 1728 | |
| 1729 | 9. If an assertion that was used as a condition was quantified with a minimum |
| 1730 | of zero, matching went wrong. In particular, if the whole group had unlimited |
| 1731 | repetition and could match an empty string, a segfault was likely. The pattern |
| 1732 | (?(?=0)?)+ is an example that caused this. Perl allows assertions to be |
| 1733 | quantified, but not if they are being used as conditions, so the above pattern |
| 1734 | is faulted by Perl. PCRE2 has now been changed so that it also rejects such |
| 1735 | patterns. |
| 1736 | |
| 1737 | 10. The error message for an invalid quantifier has been changed from "nothing |
| 1738 | to repeat" to "quantifier does not follow a repeatable item". |
| 1739 | |
| 1740 | 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but |
| 1741 | scanning the compiled pattern in subsequent auto-possessification can get out |
| 1742 | of step and lead to an unknown opcode. Previously this could have caused an |
| 1743 | infinite loop. Now it generates an "internal error" error. This is a tidyup, |
| 1744 | not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an |
| 1745 | undefined outcome. |
| 1746 | |
| 1747 | 12. A UTF pattern containing a "not" match of a non-ASCII character and a |
| 1748 | subroutine reference could loop at compile time. Example: /[^\xff]((?1))/. |
| 1749 | |
| 1750 | 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale |
| 1751 | that is found in the output of "locale -a" can actually be set by pcre2test |
| 1752 | before it is accepted. Previously, in an environment where a locale was listed |
| 1753 | but would not set (an example does exist), the test would "pass" without |
| 1754 | actually doing anything. Also the fr_CA locale has been added to the list of |
| 1755 | locales that can be used. |
| 1756 | |
| 1757 | 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a |
| 1758 | capturing group number without parentheses, the last character was incorrectly |
| 1759 | literally included at the end of the replacement string. |
| 1760 | |
| 1761 | 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero |
| 1762 | failed to allow the zero-repeat case if pcre2_match() was called with an |
| 1763 | ovector too small to capture the group. |
| 1764 | |
| 1765 | 16. Improved error message in pcre2test when setting the stack size (-S) fails. |
| 1766 | |
| 1767 | 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the |
| 1768 | transfer from PCRE1, meaning that CMake configuration failed if "build tests" |
| 1769 | was selected. (2) The file src/pcre2_serialize.c had not been added to the list |
| 1770 | of PCRE2 sources, which caused a failure to build pcre2test. |
| 1771 | |
| 1772 | 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems |
| 1773 | only on Windows. |
| 1774 | |
| 1775 | 19. Use binary input when reading back saved serialized patterns in pcre2test. |
| 1776 | |
| 1777 | 20. Added RunTest.bat for running the tests under Windows. |
| 1778 | |
| 1779 | 21. "make distclean" was not removing config.h, a file that may be created for |
| 1780 | use with CMake. |
| 1781 | |
| 1782 | 22. A pattern such as "((?2){0,1999}())?", which has a group containing a |
| 1783 | forward reference repeated a large (but limited) number of times within a |
| 1784 | repeated outer group that has a zero minimum quantifier, caused incorrect code |
| 1785 | to be compiled, leading to the error "internal error: previously-checked |
| 1786 | referenced subpattern not found" when an incorrect memory address was read. |
| 1787 | This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's |
| 1788 | FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.) |
| 1789 | |
| 1790 | 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine |
| 1791 | call within a group that also contained a recursive back reference caused |
| 1792 | incorrect code to be compiled. This bug was reported as "heap overflow", |
| 1793 | discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015: |
| 1794 | CVE-2015-2326 was given to this.) |
| 1795 | |
| 1796 | 24. Computing the size of the JIT read-only data in advance has been a source |
| 1797 | of various issues, and new ones are still appear unfortunately. To fix |
| 1798 | existing and future issues, size computation is eliminated from the code, |
| 1799 | and replaced by on-demand memory allocation. |
| 1800 | |
| 1801 | 25. A pattern such as /(?i)[A-`]/, where characters in the other case are |
| 1802 | adjacent to the end of the range, and the range contained characters with more |
| 1803 | than one other case, caused incorrect behaviour when compiled in UTF mode. In |
| 1804 | that example, the range a-j was left out of the class. |
| 1805 | |
| 1806 | |
| 1807 | Version 10.00 05-January-2015 |
| 1808 | ----------------------------- |
| 1809 | |
| 1810 | Version 10.00 is the first release of PCRE2, a revised API for the PCRE |
| 1811 | library. Changes prior to 10.00 are logged in the ChangeLog file for the old |
| 1812 | API, up to item 20 for release 8.36. |
| 1813 | |
| 1814 | The code of the library was heavily revised as part of the new API |
| 1815 | implementation. Details of each and every modification were not individually |
| 1816 | logged. In addition to the API changes, the following changes were made. They |
| 1817 | are either new functionality, or bug fixes and other noticeable changes of |
| 1818 | behaviour that were implemented after the code had been forked. |
| 1819 | |
| 1820 | 1. Including Unicode support at build time is now enabled by default, but it |
| 1821 | can optionally be disabled. It is not enabled by default at run time (no |
| 1822 | change). |
| 1823 | |
| 1824 | 2. The test program, now called pcre2test, was re-specified and almost |
| 1825 | completely re-written. Its input is not compatible with input for pcretest. |
| 1826 | |
| 1827 | 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the |
| 1828 | PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is |
| 1829 | matched by that pattern. |
| 1830 | |
| 1831 | 4. For the benefit of those who use PCRE2 via some other application, that is, |
| 1832 | not writing the function calls themselves, it is possible to check the PCRE2 |
| 1833 | version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a |
| 1834 | string such as "yesno". |
| 1835 | |
| 1836 | 5. There are case-equivalent Unicode characters whose encodings use different |
| 1837 | numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is |
| 1838 | theoretically possible for this to happen in UTF-16 too.) If a backreference to |
| 1839 | a group containing one of these characters was greedily repeated, and during |
| 1840 | the match a backtrack occurred, the subject might be backtracked by the wrong |
| 1841 | number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly |
| 1842 | (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should |
| 1843 | capture the final character, which is the three bytes E2, B1, and A5 in UTF-8. |
| 1844 | Incorrect backtracking meant that group 2 captured only the last two bytes. |
| 1845 | This bug has been fixed; the new code is slower, but it is used only when the |
| 1846 | strings matched by the repetition are not all the same length. |
| 1847 | |
| 1848 | 6. A pattern such as /()a/ was not setting the "first character must be 'a'" |
| 1849 | information. This applied to any pattern with a group that matched no |
| 1850 | characters, for example: /(?:(?=.)|(?<!x))a/. |
| 1851 | |
| 1852 | 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for |
| 1853 | those parentheses to be closed with whatever has been captured so far. However, |
| 1854 | it was failing to mark any other groups between the highest capture so far and |
| 1855 | the currrent group as "unset". Thus, the ovector for those groups contained |
| 1856 | whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when |
| 1857 | matched against "abcd". |
| 1858 | |
| 1859 | 8. The pcre2_substitute() function has been implemented. |
| 1860 | |
| 1861 | 9. If an assertion used as a condition was quantified with a minimum of zero |
| 1862 | (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could |
| 1863 | occur. |
| 1864 | |
| 1865 | 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented. |
| 1866 | |
| 1867 | **** |