1<html> 2<head> 3<title>pcre2callout specification</title> 4</head> 5<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6<h1>pcre2callout man page</h1> 7<p> 8Return to the <a href="index.html">PCRE2 index page</a>. 9</p> 10<p> 11This page is part of the PCRE2 HTML documentation. It was generated 12automatically from the original man page. If there is any nonsense in it, 13please consult the man page, in case the conversion went wrong. 14<br> 15<ul> 16<li><a name="TOC1" href="#SEC1">SYNOPSIS</a> 17<li><a name="TOC2" href="#SEC2">DESCRIPTION</a> 18<li><a name="TOC3" href="#SEC3">MISSING CALLOUTS</a> 19<li><a name="TOC4" href="#SEC4">THE CALLOUT INTERFACE</a> 20<li><a name="TOC5" href="#SEC5">RETURN VALUES FROM CALLOUTS</a> 21<li><a name="TOC6" href="#SEC6">CALLOUT ENUMERATION</a> 22<li><a name="TOC7" href="#SEC7">AUTHOR</a> 23<li><a name="TOC8" href="#SEC8">REVISION</a> 24</ul> 25<br><a name="SEC1" href="#TOC1">SYNOPSIS</a><br> 26<P> 27<b>#include <pcre2.h></b> 28</P> 29<P> 30<b>int (*pcre2_callout)(pcre2_callout_block *, void *);</b> 31<br> 32<br> 33<b>int pcre2_callout_enumerate(const pcre2_code *<i>code</i>,</b> 34<b> int (*<i>callback</i>)(pcre2_callout_enumerate_block *, void *),</b> 35<b> void *<i>user_data</i>);</b> 36</P> 37<br><a name="SEC2" href="#TOC1">DESCRIPTION</a><br> 38<P> 39PCRE2 provides a feature called "callout", which is a means of temporarily 40passing control to the caller of PCRE2 in the middle of pattern matching. The 41caller of PCRE2 provides an external function by putting its entry point in 42a match context (see <b>pcre2_set_callout()</b> in the 43<a href="pcre2api.html"><b>pcre2api</b></a> 44documentation). 45</P> 46<P> 47When using the <b>pcre2_substitute()</b> function, an additional callout feature 48is available. This does a callout after each change to the subject string and 49is described in the 50<a href="pcre2api.html"><b>pcre2api</b></a> 51documentation; the rest of this document is concerned with callouts during 52pattern matching. 53</P> 54<P> 55Within a regular expression, (?C<arg>) indicates a point at which the external 56function is to be called. Different callout points can be identified by putting 57a number less than 256 after the letter C. The default value is zero. 58Alternatively, the argument may be a delimited string. The starting delimiter 59must be one of ` ' " ^ % # $ { and the ending delimiter is the same as the 60start, except for {, where the ending delimiter is }. If the ending delimiter 61is needed within the string, it must be doubled. For example, this pattern has 62two callout points: 63<pre> 64 (?C1)abc(?C"some ""arbitrary"" text")def 65</pre> 66If the PCRE2_AUTO_CALLOUT option bit is set when a pattern is compiled, PCRE2 67automatically inserts callouts, all with number 255, before each item in the 68pattern except for immediately before or after an explicit callout. For 69example, if PCRE2_AUTO_CALLOUT is used with the pattern 70<pre> 71 A(?C3)B 72</pre> 73it is processed as if it were 74<pre> 75 (?C255)A(?C3)B(?C255) 76</pre> 77Here is a more complicated example: 78<pre> 79 A(\d{2}|--) 80</pre> 81With PCRE2_AUTO_CALLOUT, this pattern is processed as if it were 82<pre> 83 (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255) 84</pre> 85Notice that there is a callout before and after each parenthesis and 86alternation bar. If the pattern contains a conditional group whose condition is 87an assertion, an automatic callout is inserted immediately before the 88condition. Such a callout may also be inserted explicitly, for example: 89<pre> 90 (?(?C9)(?=a)ab|de) (?(?C%text%)(?!=d)ab|de) 91</pre> 92This applies only to assertion conditions (because they are themselves 93independent groups). 94</P> 95<P> 96Callouts can be useful for tracking the progress of pattern matching. The 97<a href="pcre2test.html"><b>pcre2test</b></a> 98program has a pattern qualifier (/auto_callout) that sets automatic callouts. 99When any callouts are present, the output from <b>pcre2test</b> indicates how 100the pattern is being matched. This is useful information when you are trying to 101optimize the performance of a particular pattern. 102</P> 103<br><a name="SEC3" href="#TOC1">MISSING CALLOUTS</a><br> 104<P> 105You should be aware that, because of optimizations in the way PCRE2 compiles 106and matches patterns, callouts sometimes do not happen exactly as you might 107expect. 108</P> 109<br><b> 110Auto-possessification 111</b><br> 112<P> 113At compile time, PCRE2 "auto-possessifies" repeated items when it knows that 114what follows cannot be part of the repeat. For example, a+[bc] is compiled as 115if it were a++[bc]. The <b>pcre2test</b> output when this pattern is compiled 116with PCRE2_ANCHORED and PCRE2_AUTO_CALLOUT and then applied to the string 117"aaaa" is: 118<pre> 119 --->aaaa 120 +0 ^ a+ 121 +2 ^ ^ [bc] 122 No match 123</pre> 124This indicates that when matching [bc] fails, there is no backtracking into a+ 125(because it is being treated as a++) and therefore the callouts that would be 126taken for the backtracks do not occur. You can disable the auto-possessify 127feature by passing PCRE2_NO_AUTO_POSSESS to <b>pcre2_compile()</b>, or starting 128the pattern with (*NO_AUTO_POSSESS). In this case, the output changes to this: 129<pre> 130 --->aaaa 131 +0 ^ a+ 132 +2 ^ ^ [bc] 133 +2 ^ ^ [bc] 134 +2 ^ ^ [bc] 135 +2 ^^ [bc] 136 No match 137</pre> 138This time, when matching [bc] fails, the matcher backtracks into a+ and tries 139again, repeatedly, until a+ itself fails. 140</P> 141<br><b> 142Automatic .* anchoring 143</b><br> 144<P> 145By default, an optimization is applied when .* is the first significant item in 146a pattern. If PCRE2_DOTALL is set, so that the dot can match any character, the 147pattern is automatically anchored. If PCRE2_DOTALL is not set, a match can 148start only after an internal newline or at the beginning of the subject, and 149<b>pcre2_compile()</b> remembers this. If a pattern has more than one top-level 150branch, automatic anchoring occurs if all branches are anchorable. 151</P> 152<P> 153This optimization is disabled, however, if .* is in an atomic group or if there 154is a backreference to the capture group in which it appears. It is also 155disabled if the pattern contains (*PRUNE) or (*SKIP). However, the presence of 156callouts does not affect it. 157</P> 158<P> 159For example, if the pattern .*\d is compiled with PCRE2_AUTO_CALLOUT and 160applied to the string "aa", the <b>pcre2test</b> output is: 161<pre> 162 --->aa 163 +0 ^ .* 164 +2 ^ ^ \d 165 +2 ^^ \d 166 +2 ^ \d 167 No match 168</pre> 169This shows that all match attempts start at the beginning of the subject. In 170other words, the pattern is anchored. You can disable this optimization by 171passing PCRE2_NO_DOTSTAR_ANCHOR to <b>pcre2_compile()</b>, or starting the 172pattern with (*NO_DOTSTAR_ANCHOR). In this case, the output changes to: 173<pre> 174 --->aa 175 +0 ^ .* 176 +2 ^ ^ \d 177 +2 ^^ \d 178 +2 ^ \d 179 +0 ^ .* 180 +2 ^^ \d 181 +2 ^ \d 182 No match 183</pre> 184This shows more match attempts, starting at the second subject character. 185Another optimization, described in the next section, means that there is no 186subsequent attempt to match with an empty subject. 187</P> 188<br><b> 189Other optimizations 190</b><br> 191<P> 192Other optimizations that provide fast "no match" results also affect callouts. 193For example, if the pattern is 194<pre> 195 ab(?C4)cd 196</pre> 197PCRE2 knows that any matching string must contain the letter "d". If the 198subject string is "abyz", the lack of "d" means that matching doesn't ever 199start, and the callout is never reached. However, with "abyd", though the 200result is still no match, the callout is obeyed. 201</P> 202<P> 203For most patterns PCRE2 also knows the minimum length of a matching string, and 204will immediately give a "no match" return without actually running a match if 205the subject is not long enough, or, for unanchored patterns, if it has been 206scanned far enough. 207</P> 208<P> 209You can disable these optimizations by passing the PCRE2_NO_START_OPTIMIZE 210option to <b>pcre2_compile()</b>, or by starting the pattern with 211(*NO_START_OPT). This slows down the matching process, but does ensure that 212callouts such as the example above are obeyed. 213<a name="calloutinterface"></a></P> 214<br><a name="SEC4" href="#TOC1">THE CALLOUT INTERFACE</a><br> 215<P> 216During matching, when PCRE2 reaches a callout point, if an external function is 217provided in the match context, it is called. This applies to both normal, 218DFA, and JIT matching. The first argument to the callout function is a pointer 219to a <b>pcre2_callout</b> block. The second argument is the void * callout data 220that was supplied when the callout was set up by calling 221<b>pcre2_set_callout()</b> (see the 222<a href="pcre2api.html"><b>pcre2api</b></a> 223documentation). The callout block structure contains the following fields, not 224necessarily in this order: 225<pre> 226 uint32_t <i>version</i>; 227 uint32_t <i>callout_number</i>; 228 uint32_t <i>capture_top</i>; 229 uint32_t <i>capture_last</i>; 230 uint32_t <i>callout_flags</i>; 231 PCRE2_SIZE *<i>offset_vector</i>; 232 PCRE2_SPTR <i>mark</i>; 233 PCRE2_SPTR <i>subject</i>; 234 PCRE2_SIZE <i>subject_length</i>; 235 PCRE2_SIZE <i>start_match</i>; 236 PCRE2_SIZE <i>current_position</i>; 237 PCRE2_SIZE <i>pattern_position</i>; 238 PCRE2_SIZE <i>next_item_length</i>; 239 PCRE2_SIZE <i>callout_string_offset</i>; 240 PCRE2_SIZE <i>callout_string_length</i>; 241 PCRE2_SPTR <i>callout_string</i>; 242</pre> 243The <i>version</i> field contains the version number of the block format. The 244current version is 2; the three callout string fields were added for version 1, 245and the <i>callout_flags</i> field for version 2. If you are writing an 246application that might use an earlier release of PCRE2, you should check the 247version number before accessing any of these fields. The version number will 248increase in future if more fields are added, but the intention is never to 249remove any of the existing fields. 250</P> 251<br><b> 252Fields for numerical callouts 253</b><br> 254<P> 255For a numerical callout, <i>callout_string</i> is NULL, and <i>callout_number</i> 256contains the number of the callout, in the range 0-255. This is the number 257that follows (?C for callouts that part of the pattern; it is 255 for 258automatically generated callouts. 259</P> 260<br><b> 261Fields for string callouts 262</b><br> 263<P> 264For callouts with string arguments, <i>callout_number</i> is always zero, and 265<i>callout_string</i> points to the string that is contained within the compiled 266pattern. Its length is given by <i>callout_string_length</i>. Duplicated ending 267delimiters that were present in the original pattern string have been turned 268into single characters, but there is no other processing of the callout string 269argument. An additional code unit containing binary zero is present after the 270string, but is not included in the length. The delimiter that was used to start 271the string is also stored within the pattern, immediately before the string 272itself. You can access this delimiter as <i>callout_string</i>[-1] if you need 273it. 274</P> 275<P> 276The <i>callout_string_offset</i> field is the code unit offset to the start of 277the callout argument string within the original pattern string. This is 278provided for the benefit of applications such as script languages that might 279need to report errors in the callout string within the pattern. 280</P> 281<br><b> 282Fields for all callouts 283</b><br> 284<P> 285The remaining fields in the callout block are the same for both kinds of 286callout. 287</P> 288<P> 289The <i>offset_vector</i> field is a pointer to a vector of capturing offsets 290(the "ovector"). You may read the elements in this vector, but you must not 291change any of them. 292</P> 293<P> 294For calls to <b>pcre2_match()</b>, the <i>offset_vector</i> field is not (since 295release 10.30) a pointer to the actual ovector that was passed to the matching 296function in the match data block. Instead it points to an internal ovector of a 297size large enough to hold all possible captured substrings in the pattern. Note 298that whenever a recursion or subroutine call within a pattern completes, the 299capturing state is reset to what it was before. 300</P> 301<P> 302The <i>capture_last</i> field contains the number of the most recently captured 303substring, and the <i>capture_top</i> field contains one more than the number of 304the highest numbered captured substring so far. If no substrings have yet been 305captured, the value of <i>capture_last</i> is 0 and the value of 306<i>capture_top</i> is 1. The values of these fields do not always differ by one; 307for example, when the callout in the pattern ((a)(b))(?C2) is taken, 308<i>capture_last</i> is 1 but <i>capture_top</i> is 4. 309</P> 310<P> 311The contents of ovector[2] to ovector[<capture_top>*2-1] can be inspected in 312order to extract substrings that have been matched so far, in the same way as 313extracting substrings after a match has completed. The values in ovector[0] and 314ovector[1] are always PCRE2_UNSET because the match is by definition not 315complete. Substrings that have not been captured but whose numbers are less 316than <i>capture_top</i> also have both of their ovector slots set to 317PCRE2_UNSET. 318</P> 319<P> 320For DFA matching, the <i>offset_vector</i> field points to the ovector that was 321passed to the matching function in the match data block for callouts at the top 322level, but to an internal ovector during the processing of pattern recursions, 323lookarounds, and atomic groups. However, these ovectors hold no useful 324information because <b>pcre2_dfa_match()</b> does not support substring 325capturing. The value of <i>capture_top</i> is always 1 and the value of 326<i>capture_last</i> is always 0 for DFA matching. 327</P> 328<P> 329The <i>subject</i> and <i>subject_length</i> fields contain copies of the values 330that were passed to the matching function. 331</P> 332<P> 333The <i>start_match</i> field normally contains the offset within the subject at 334which the current match attempt started. However, if the escape sequence \K 335has been encountered, this value is changed to reflect the modified starting 336point. If the pattern is not anchored, the callout function may be called 337several times from the same point in the pattern for different starting points 338in the subject. 339</P> 340<P> 341The <i>current_position</i> field contains the offset within the subject of the 342current match pointer. 343</P> 344<P> 345The <i>pattern_position</i> field contains the offset in the pattern string to 346the next item to be matched. 347</P> 348<P> 349The <i>next_item_length</i> field contains the length of the next item to be 350processed in the pattern string. When the callout is at the end of the pattern, 351the length is zero. When the callout precedes an opening parenthesis, the 352length includes meta characters that follow the parenthesis. For example, in a 353callout before an assertion such as (?=ab) the length is 3. For an alternation 354bar or a closing parenthesis, the length is one, unless a closing parenthesis 355is followed by a quantifier, in which case its length is included. (This 356changed in release 10.23. In earlier releases, before an opening parenthesis 357the length was that of the entire group, and before an alternation bar or a 358closing parenthesis the length was zero.) 359</P> 360<P> 361The <i>pattern_position</i> and <i>next_item_length</i> fields are intended to 362help in distinguishing between different automatic callouts, which all have the 363same callout number. However, they are set for all callouts, and are used by 364<b>pcre2test</b> to show the next item to be matched when displaying callout 365information. 366</P> 367<P> 368In callouts from <b>pcre2_match()</b> the <i>mark</i> field contains a pointer to 369the zero-terminated name of the most recently passed (*MARK), (*PRUNE), or 370(*THEN) item in the match, or NULL if no such items have been passed. Instances 371of (*PRUNE) or (*THEN) without a name do not obliterate a previous (*MARK). In 372callouts from the DFA matching function this field always contains NULL. 373</P> 374<P> 375The <i>callout_flags</i> field is always zero in callouts from 376<b>pcre2_dfa_match()</b> or when JIT is being used. When <b>pcre2_match()</b> 377without JIT is used, the following bits may be set: 378<pre> 379 PCRE2_CALLOUT_STARTMATCH 380</pre> 381This is set for the first callout after the start of matching for each new 382starting position in the subject. 383<pre> 384 PCRE2_CALLOUT_BACKTRACK 385</pre> 386This is set if there has been a matching backtrack since the previous callout, 387or since the start of matching if this is the first callout from a 388<b>pcre2_match()</b> run. 389</P> 390<P> 391Both bits are set when a backtrack has caused a "bumpalong" to a new starting 392position in the subject. Output from <b>pcre2test</b> does not indicate the 393presence of these bits unless the <b>callout_extra</b> modifier is set. 394</P> 395<P> 396The information in the <b>callout_flags</b> field is provided so that 397applications can track and tell their users how matching with backtracking is 398done. This can be useful when trying to optimize patterns, or just to 399understand how PCRE2 works. There is no support in <b>pcre2_dfa_match()</b> 400because there is no backtracking in DFA matching, and there is no support in 401JIT because JIT is all about maximimizing matching performance. In both these 402cases the <b>callout_flags</b> field is always zero. 403</P> 404<br><a name="SEC5" href="#TOC1">RETURN VALUES FROM CALLOUTS</a><br> 405<P> 406The external callout function returns an integer to PCRE2. If the value is 407zero, matching proceeds as normal. If the value is greater than zero, matching 408fails at the current point, but the testing of other matching possibilities 409goes ahead, just as if a lookahead assertion had failed. If the value is less 410than zero, the match is abandoned, and the matching function returns the 411negative value. 412</P> 413<P> 414Negative values should normally be chosen from the set of PCRE2_ERROR_xxx 415values. In particular, PCRE2_ERROR_NOMATCH forces a standard "no match" 416failure. The error number PCRE2_ERROR_CALLOUT is reserved for use by callout 417functions; it will never be used by PCRE2 itself. 418</P> 419<br><a name="SEC6" href="#TOC1">CALLOUT ENUMERATION</a><br> 420<P> 421<b>int pcre2_callout_enumerate(const pcre2_code *<i>code</i>,</b> 422<b> int (*<i>callback</i>)(pcre2_callout_enumerate_block *, void *),</b> 423<b> void *<i>user_data</i>);</b> 424<br> 425<br> 426A script language that supports the use of string arguments in callouts might 427like to scan all the callouts in a pattern before running the match. This can 428be done by calling <b>pcre2_callout_enumerate()</b>. The first argument is a 429pointer to a compiled pattern, the second points to a callback function, and 430the third is arbitrary user data. The callback function is called for every 431callout in the pattern in the order in which they appear. Its first argument is 432a pointer to a callout enumeration block, and its second argument is the 433<i>user_data</i> value that was passed to <b>pcre2_callout_enumerate()</b>. The 434data block contains the following fields: 435<pre> 436 <i>version</i> Block version number 437 <i>pattern_position</i> Offset to next item in pattern 438 <i>next_item_length</i> Length of next item in pattern 439 <i>callout_number</i> Number for numbered callouts 440 <i>callout_string_offset</i> Offset to string within pattern 441 <i>callout_string_length</i> Length of callout string 442 <i>callout_string</i> Points to callout string or is NULL 443</pre> 444The version number is currently 0. It will increase if new fields are ever 445added to the block. The remaining fields are the same as their namesakes in the 446<b>pcre2_callout</b> block that is used for callouts during matching, as 447described 448<a href="#calloutinterface">above.</a> 449</P> 450<P> 451Note that the value of <i>pattern_position</i> is unique for each callout. 452However, if a callout occurs inside a group that is quantified with a non-zero 453minimum or a fixed maximum, the group is replicated inside the compiled 454pattern. For example, a pattern such as /(a){2}/ is compiled as if it were 455/(a)(a)/. This means that the callout will be enumerated more than once, but 456with the same value for <i>pattern_position</i> in each case. 457</P> 458<P> 459The callback function should normally return zero. If it returns a non-zero 460value, scanning the pattern stops, and that value is returned from 461<b>pcre2_callout_enumerate()</b>. 462</P> 463<br><a name="SEC7" href="#TOC1">AUTHOR</a><br> 464<P> 465Philip Hazel 466<br> 467Retired from University Computing Service 468<br> 469Cambridge, England. 470<br> 471</P> 472<br><a name="SEC8" href="#TOC1">REVISION</a><br> 473<P> 474Last updated: 19 January 2024 475<br> 476Copyright © 1997-2024 University of Cambridge. 477<br> 478<p> 479Return to the <a href="index.html">PCRE2 index page</a>. 480</p> 481