|
1 <html> |
|
2 <head> |
|
3 <title>pcre specification</title> |
|
4 </head> |
|
5 <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> |
|
6 <h1>pcre man page</h1> |
|
7 <p> |
|
8 Return to the <a href="index.html">PCRE index page</a>. |
|
9 </p> |
|
10 <p> |
|
11 This page is part of the PCRE HTML documentation. It was generated automatically |
|
12 from the original man page. If there is any nonsense in it, please consult the |
|
13 man page, in case the conversion went wrong. |
|
14 <br> |
|
15 <ul> |
|
16 <li><a name="TOC1" href="#SEC1">INTRODUCTION</a> |
|
17 <li><a name="TOC2" href="#SEC2">USER DOCUMENTATION</a> |
|
18 <li><a name="TOC3" href="#SEC3">LIMITATIONS</a> |
|
19 <li><a name="TOC4" href="#SEC4">UTF-8 AND UNICODE PROPERTY SUPPORT</a> |
|
20 <li><a name="TOC5" href="#SEC5">AUTHOR</a> |
|
21 <li><a name="TOC6" href="#SEC6">REVISION</a> |
|
22 </ul> |
|
23 <br><a name="SEC1" href="#TOC1">INTRODUCTION</a><br> |
|
24 <P> |
|
25 The PCRE library is a set of functions that implement regular expression |
|
26 pattern matching using the same syntax and semantics as Perl, with just a few |
|
27 differences. Certain features that appeared in Python and PCRE before they |
|
28 appeared in Perl are also available using the Python syntax. There is also some |
|
29 support for certain .NET and Oniguruma syntax items, and there is an option for |
|
30 requesting some minor changes that give better JavaScript compatibility. |
|
31 </P> |
|
32 <P> |
|
33 The current implementation of PCRE (release 7.x) corresponds approximately with |
|
34 Perl 5.10, including support for UTF-8 encoded strings and Unicode general |
|
35 category properties. However, UTF-8 and Unicode support has to be explicitly |
|
36 enabled; it is not the default. The Unicode tables correspond to Unicode |
|
37 release 5.0.0. |
|
38 </P> |
|
39 <P> |
|
40 In addition to the Perl-compatible matching function, PCRE contains an |
|
41 alternative matching function that matches the same compiled patterns in a |
|
42 different way. In certain circumstances, the alternative function has some |
|
43 advantages. For a discussion of the two matching algorithms, see the |
|
44 <a href="pcrematching.html"><b>pcrematching</b></a> |
|
45 page. |
|
46 </P> |
|
47 <P> |
|
48 PCRE is written in C and released as a C library. A number of people have |
|
49 written wrappers and interfaces of various kinds. In particular, Google Inc. |
|
50 have provided a comprehensive C++ wrapper. This is now included as part of the |
|
51 PCRE distribution. The |
|
52 <a href="pcrecpp.html"><b>pcrecpp</b></a> |
|
53 page has details of this interface. Other people's contributions can be found |
|
54 in the <i>Contrib</i> directory at the primary FTP site, which is: |
|
55 <a href="ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre">ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre</a> |
|
56 </P> |
|
57 <P> |
|
58 Details of exactly which Perl regular expression features are and are not |
|
59 supported by PCRE are given in separate documents. See the |
|
60 <a href="pcrepattern.html"><b>pcrepattern</b></a> |
|
61 and |
|
62 <a href="pcrecompat.html"><b>pcrecompat</b></a> |
|
63 pages. There is a syntax summary in the |
|
64 <a href="pcresyntax.html"><b>pcresyntax</b></a> |
|
65 page. |
|
66 </P> |
|
67 <P> |
|
68 Some features of PCRE can be included, excluded, or changed when the library is |
|
69 built. The |
|
70 <a href="pcre_config.html"><b>pcre_config()</b></a> |
|
71 function makes it possible for a client to discover which features are |
|
72 available. The features themselves are described in the |
|
73 <a href="pcrebuild.html"><b>pcrebuild</b></a> |
|
74 page. Documentation about building PCRE for various operating systems can be |
|
75 found in the <b>README</b> file in the source distribution. |
|
76 </P> |
|
77 <P> |
|
78 The library contains a number of undocumented internal functions and data |
|
79 tables that are used by more than one of the exported external functions, but |
|
80 which are not intended for use by external callers. Their names all begin with |
|
81 "_pcre_", which hopefully will not provoke any name clashes. In some |
|
82 environments, it is possible to control which external symbols are exported |
|
83 when a shared library is built, and in these cases the undocumented symbols are |
|
84 not exported. |
|
85 </P> |
|
86 <br><a name="SEC2" href="#TOC1">USER DOCUMENTATION</a><br> |
|
87 <P> |
|
88 The user documentation for PCRE comprises a number of different sections. In |
|
89 the "man" format, each of these is a separate "man page". In the HTML format, |
|
90 each is a separate page, linked from the index page. In the plain text format, |
|
91 all the sections are concatenated, for ease of searching. The sections are as |
|
92 follows: |
|
93 <pre> |
|
94 pcre this document |
|
95 pcre-config show PCRE installation configuration information |
|
96 pcreapi details of PCRE's native C API |
|
97 pcrebuild options for building PCRE |
|
98 pcrecallout details of the callout feature |
|
99 pcrecompat discussion of Perl compatibility |
|
100 pcrecpp details of the C++ wrapper |
|
101 pcregrep description of the <b>pcregrep</b> command |
|
102 pcrematching discussion of the two matching algorithms |
|
103 pcrepartial details of the partial matching facility |
|
104 pcrepattern syntax and semantics of supported regular expressions |
|
105 pcresyntax quick syntax reference |
|
106 pcreperform discussion of performance issues |
|
107 pcreposix the POSIX-compatible C API |
|
108 pcreprecompile details of saving and re-using precompiled patterns |
|
109 pcresample discussion of the sample program |
|
110 pcrestack discussion of stack usage |
|
111 pcretest description of the <b>pcretest</b> testing command |
|
112 </pre> |
|
113 In addition, in the "man" and HTML formats, there is a short page for each |
|
114 C library function, listing its arguments and results. |
|
115 </P> |
|
116 <br><a name="SEC3" href="#TOC1">LIMITATIONS</a><br> |
|
117 <P> |
|
118 There are some size limitations in PCRE but it is hoped that they will never in |
|
119 practice be relevant. |
|
120 </P> |
|
121 <P> |
|
122 The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE is |
|
123 compiled with the default internal linkage size of 2. If you want to process |
|
124 regular expressions that are truly enormous, you can compile PCRE with an |
|
125 internal linkage size of 3 or 4 (see the <b>README</b> file in the source |
|
126 distribution and the |
|
127 <a href="pcrebuild.html"><b>pcrebuild</b></a> |
|
128 documentation for details). In these cases the limit is substantially larger. |
|
129 However, the speed of execution is slower. |
|
130 </P> |
|
131 <P> |
|
132 All values in repeating quantifiers must be less than 65536. |
|
133 </P> |
|
134 <P> |
|
135 There is no limit to the number of parenthesized subpatterns, but there can be |
|
136 no more than 65535 capturing subpatterns. |
|
137 </P> |
|
138 <P> |
|
139 The maximum length of name for a named subpattern is 32 characters, and the |
|
140 maximum number of named subpatterns is 10000. |
|
141 </P> |
|
142 <P> |
|
143 The maximum length of a subject string is the largest positive number that an |
|
144 integer variable can hold. However, when using the traditional matching |
|
145 function, PCRE uses recursion to handle subpatterns and indefinite repetition. |
|
146 This means that the available stack space may limit the size of a subject |
|
147 string that can be processed by certain patterns. For a discussion of stack |
|
148 issues, see the |
|
149 <a href="pcrestack.html"><b>pcrestack</b></a> |
|
150 documentation. |
|
151 <a name="utf8support"></a></P> |
|
152 <br><a name="SEC4" href="#TOC1">UTF-8 AND UNICODE PROPERTY SUPPORT</a><br> |
|
153 <P> |
|
154 From release 3.3, PCRE has had some support for character strings encoded in |
|
155 the UTF-8 format. For release 4.0 this was greatly extended to cover most |
|
156 common requirements, and in release 5.0 additional support for Unicode general |
|
157 category properties was added. |
|
158 </P> |
|
159 <P> |
|
160 In order process UTF-8 strings, you must build PCRE to include UTF-8 support in |
|
161 the code, and, in addition, you must call |
|
162 <a href="pcre_compile.html"><b>pcre_compile()</b></a> |
|
163 with the PCRE_UTF8 option flag. When you do this, both the pattern and any |
|
164 subject strings that are matched against it are treated as UTF-8 strings |
|
165 instead of just strings of bytes. |
|
166 </P> |
|
167 <P> |
|
168 If you compile PCRE with UTF-8 support, but do not use it at run time, the |
|
169 library will be a bit bigger, but the additional run time overhead is limited |
|
170 to testing the PCRE_UTF8 flag occasionally, so should not be very big. |
|
171 </P> |
|
172 <P> |
|
173 If PCRE is built with Unicode character property support (which implies UTF-8 |
|
174 support), the escape sequences \p{..}, \P{..}, and \X are supported. |
|
175 The available properties that can be tested are limited to the general |
|
176 category properties such as Lu for an upper case letter or Nd for a decimal |
|
177 number, the Unicode script names such as Arabic or Han, and the derived |
|
178 properties Any and L&. A full list is given in the |
|
179 <a href="pcrepattern.html"><b>pcrepattern</b></a> |
|
180 documentation. Only the short names for properties are supported. For example, |
|
181 \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported. |
|
182 Furthermore, in Perl, many properties may optionally be prefixed by "Is", for |
|
183 compatibility with Perl 5.6. PCRE does not support this. |
|
184 <a name="utf8strings"></a></P> |
|
185 <br><b> |
|
186 Validity of UTF-8 strings |
|
187 </b><br> |
|
188 <P> |
|
189 When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects |
|
190 are (by default) checked for validity on entry to the relevant functions. From |
|
191 release 7.3 of PCRE, the check is according the rules of RFC 3629, which are |
|
192 themselves derived from the Unicode specification. Earlier releases of PCRE |
|
193 followed the rules of RFC 2279, which allows the full range of 31-bit values (0 |
|
194 to 0x7FFFFFFF). The current check allows only values in the range U+0 to |
|
195 U+10FFFF, excluding U+D800 to U+DFFF. |
|
196 </P> |
|
197 <P> |
|
198 The excluded code points are the "Low Surrogate Area" of Unicode, of which the |
|
199 Unicode Standard says this: "The Low Surrogate Area does not contain any |
|
200 character assignments, consequently no character code charts or namelists are |
|
201 provided for this area. Surrogates are reserved for use with UTF-16 and then |
|
202 must be used in pairs." The code points that are encoded by UTF-16 pairs are |
|
203 available as independent code points in the UTF-8 encoding. (In other words, |
|
204 the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up |
|
205 UTF-8.) |
|
206 </P> |
|
207 <P> |
|
208 If an invalid UTF-8 string is passed to PCRE, an error return |
|
209 (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know that |
|
210 your strings are valid, and therefore want to skip these checks in order to |
|
211 improve performance. If you set the PCRE_NO_UTF8_CHECK flag at compile time or |
|
212 at run time, PCRE assumes that the pattern or subject it is given |
|
213 (respectively) contains only valid UTF-8 codes. In this case, it does not |
|
214 diagnose an invalid UTF-8 string. |
|
215 </P> |
|
216 <P> |
|
217 If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what |
|
218 happens depends on why the string is invalid. If the string conforms to the |
|
219 "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters |
|
220 in the range 0 to 0x7FFFFFFF. In other words, apart from the initial validity |
|
221 test, PCRE (when in UTF-8 mode) handles strings according to the more liberal |
|
222 rules of RFC 2279. However, if the string does not even conform to RFC 2279, |
|
223 the result is undefined. Your program may crash. |
|
224 </P> |
|
225 <P> |
|
226 If you want to process strings of values in the full range 0 to 0x7FFFFFFF, |
|
227 encoded in a UTF-8-like manner as per the old RFC, you can set |
|
228 PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this |
|
229 situation, you will have to apply your own validity check. |
|
230 </P> |
|
231 <br><b> |
|
232 General comments about UTF-8 mode |
|
233 </b><br> |
|
234 <P> |
|
235 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a two-byte |
|
236 UTF-8 character if the value is greater than 127. |
|
237 </P> |
|
238 <P> |
|
239 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8 |
|
240 characters for values greater than \177. |
|
241 </P> |
|
242 <P> |
|
243 3. Repeat quantifiers apply to complete UTF-8 characters, not to individual |
|
244 bytes, for example: \x{100}{3}. |
|
245 </P> |
|
246 <P> |
|
247 4. The dot metacharacter matches one UTF-8 character instead of a single byte. |
|
248 </P> |
|
249 <P> |
|
250 5. The escape sequence \C can be used to match a single byte in UTF-8 mode, |
|
251 but its use can lead to some strange effects. This facility is not available in |
|
252 the alternative matching function, <b>pcre_dfa_exec()</b>. |
|
253 </P> |
|
254 <P> |
|
255 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly |
|
256 test characters of any code value, but the characters that PCRE recognizes as |
|
257 digits, spaces, or word characters remain the same set as before, all with |
|
258 values less than 256. This remains true even when PCRE includes Unicode |
|
259 property support, because to do otherwise would slow down PCRE in many common |
|
260 cases. If you really want to test for a wider sense of, say, "digit", you |
|
261 must use Unicode property tests such as \p{Nd}. |
|
262 </P> |
|
263 <P> |
|
264 7. Similarly, characters that match the POSIX named character classes are all |
|
265 low-valued characters. |
|
266 </P> |
|
267 <P> |
|
268 8. However, the Perl 5.10 horizontal and vertical whitespace matching escapes |
|
269 (\h, \H, \v, and \V) do match all the appropriate Unicode characters. |
|
270 </P> |
|
271 <P> |
|
272 9. Case-insensitive matching applies only to characters whose values are less |
|
273 than 128, unless PCRE is built with Unicode property support. Even when Unicode |
|
274 property support is available, PCRE still uses its own character tables when |
|
275 checking the case of low-valued characters, so as not to degrade performance. |
|
276 The Unicode property information is used only for characters with higher |
|
277 values. Even when Unicode property support is available, PCRE supports |
|
278 case-insensitive matching only when there is a one-to-one mapping between a |
|
279 letter's cases. There are a small number of many-to-one mappings in Unicode; |
|
280 these are not supported by PCRE. |
|
281 </P> |
|
282 <br><a name="SEC5" href="#TOC1">AUTHOR</a><br> |
|
283 <P> |
|
284 Philip Hazel |
|
285 <br> |
|
286 University Computing Service |
|
287 <br> |
|
288 Cambridge CB2 3QH, England. |
|
289 <br> |
|
290 </P> |
|
291 <P> |
|
292 Putting an actual email address here seems to have been a spam magnet, so I've |
|
293 taken it away. If you want to email me, use my two initials, followed by the |
|
294 two digits 10, at the domain cam.ac.uk. |
|
295 </P> |
|
296 <br><a name="SEC6" href="#TOC1">REVISION</a><br> |
|
297 <P> |
|
298 Last updated: 12 April 2008 |
|
299 <br> |
|
300 Copyright © 1997-2008 University of Cambridge. |
|
301 <br> |
|
302 <p> |
|
303 Return to the <a href="index.html">PCRE index page</a>. |
|
304 </p> |