|
1 <?xml version="1.0" encoding="utf-8"?> |
|
2 <!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. --> |
|
3 <!-- This component and the accompanying materials are made available under the terms of the License |
|
4 "Eclipse Public License v1.0" which accompanies this distribution, |
|
5 and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". --> |
|
6 <!-- Initial Contributors: |
|
7 Nokia Corporation - initial contribution. |
|
8 Contributors: |
|
9 --> |
|
10 <!DOCTYPE concept |
|
11 PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
|
12 <concept id="GUID-D86B83B5-298E-510A-B984-7307CCA1AE1C" xml:lang="en"><title>How to |
|
13 use Lexical Analysis</title><shortdesc>Explains how to create and use TLex objects with code fragments.</shortdesc><prolog><metadata><keywords/></metadata></prolog><conbody> |
|
14 <section id="GUID-8A379959-AD24-4CA7-8664-63FDDBFA2A6B"><title>Introduction</title> <p>A <codeph>TLex</codeph> object may |
|
15 be constructed as an empty object, from another <codeph>TLex,</codeph> or |
|
16 from an existing string as below. The code fragments are taken from an example |
|
17 that implements a simple Reverse Polish Notation calculation engine.</p> <codeblock id="GUID-0F57D0AE-19C2-5C5D-BF05-5924643A933E" xml:space="preserve">TInt RPNCalc(const TDesC& aCommand, TReal& aReturnValue) |
|
18 { |
|
19 ... |
|
20 TLex input (aCommand) ; |
|
21 ...</codeblock> <p>Code can then proceed to move through the <codeph>TLex</codeph> data |
|
22 to:</p> <ul> |
|
23 <li id="GUID-DE9D6065-0426-5D33-A11B-3DEB3102A1EA"><p>mark positions for rolling |
|
24 back to later</p> </li> |
|
25 <li id="GUID-915C7EE1-AF85-5865-BD9A-7068426BC897"><p> delimit the start of |
|
26 lexical tokens</p> </li> |
|
27 <li id="GUID-0A43CFC8-B38B-5A94-AFA0-2C54266905F4"><p>delete parts of the |
|
28 string held by the <codeph>TLex</codeph> object.</p> </li> |
|
29 </ul> <codeblock id="GUID-2C636982-12F5-5DF6-A375-CCE4BD31FCA6" xml:space="preserve">... |
|
30 input.Mark() ; // Remember where we are. |
|
31 input.SkipCharacters() ; // Move to end of character token. |
|
32 ... |
|
33 _LIT(KTextMemset,"MEMSET"); |
|
34 if ( input.TokenLength() != 0 ) // if valid potential token |
|
35 { |
|
36 TPtrC token = input.MarkedToken() ; // then extract token |
|
37 if ( token.CompareF(KTextMemset == 0) // and test. |
|
38 { |
|
39 ... |
|
40 } |
|
41 ... |
|
42 } |
|
43 ...</codeblock> <p>Analysis can also be done by character using functions |
|
44 that move through the <codeph>TLex</codeph> data, extracting, returning and |
|
45 jumping specified character lengths. For example:</p> <codeblock id="GUID-E5650436-99E6-56C9-9CEF-2DF87F1D787D" xml:space="preserve">... |
|
46 // ensure we are looking at a digit or sign |
|
47 if (!(input.Peek().IsDigit() || (input.Peek() == '.') ) ) |
|
48 { |
|
49 return KErrNotFound ; |
|
50 } |
|
51 ... |
|
52 // deal with sign |
|
53 if (input.Peek() == '+') |
|
54 { |
|
55 input.Inc(); |
|
56 }</codeblock> <p>Additionally, numeric conversion functions permit a variety |
|
57 of numeric formats to be extracted from the <codeph>TLex</codeph> data, with |
|
58 provision for conversion using the most common number systems (radixes).</p> <codeblock id="GUID-BBE6F232-7E65-58A0-BDC8-F0293E1773F7" xml:space="preserve">... |
|
59 if (input.Val(extractUint) == KErrNone) |
|
60 { |
|
61 stack.Push(TReal(extractUint)); |
|
62 } |
|
63 else if (input.Val(extractReal) == KErrNone) |
|
64 { |
|
65 stack.Push(extractReal); |
|
66 } |
|
67 ... |
|
68 |
|
69 </codeblock> <p>where <codeph>stack</codeph>, is an instance of a class implementing |
|
70 a stack.</p> </section> |
|
71 <section id="GUID-F688DD13-C866-5752-8530-30DECBFF0054"><title>Constructing |
|
72 TLex objects</title> <p>This converts an real number into a string, which |
|
73 is then assigned to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-CB88572D-66DC-5C54-9F74-02E9331A7BAD" xml:space="preserve">TBuf<0x100> convertRealToString; |
|
74 // want a TLex from a value |
|
75 if (convertRealToString.Num(value,format) < KErrNone ) |
|
76 { |
|
77 ... |
|
78 } |
|
79 else |
|
80 { |
|
81 convertRealToString.ZeroTerminate(); |
|
82 TLex string(convertRealToString) ; |
|
83 }</codeblock> <p>This takes a descriptor as a function parameter, and |
|
84 copies it to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-B793E5B1-76A5-5178-B1E1-A569EA3C55EE" xml:space="preserve">TInt RPNCalc(const TDesC& iCommand, TReal& returnValue) |
|
85 { |
|
86 TLex input (iCommand) ; |
|
87 }</codeblock> </section> |
|
88 <section id="GUID-C5B343D9-C530-5FA9-BED9-286C182081CF"><title>Peeking the |
|
89 next character</title> <p>This shows a code flow decision made according to |
|
90 next character to be read from the <codeph>TLex</codeph>:</p> <codeblock id="GUID-6FDB6BC3-9AB7-5C04-A003-7EE0CD26F396" xml:space="preserve">if (!(input.Peek()).IsDigit()) // found non-digit after decimal point if </codeblock> </section> |
|
91 <section id="GUID-B08D3122-C580-586F-8D62-75E75A1DC717"><title>Moving past |
|
92 a character that has been peeked</title> <p>This shows the use of the <codeph>Inc()</codeph> function |
|
93 to move past a character that has been peeked:</p> <codeblock id="GUID-F2F1A3B4-C42E-56F0-BA5C-2F9391CCD2FB" xml:space="preserve">if (input.Peek() == '-') |
|
94 { |
|
95 input.Inc() ; // move past minus sign & flag |
|
96 negative = ETrue ; |
|
97 }</codeblock> </section> |
|
98 <section id="GUID-BF5B871F-CAFF-50A9-8833-A79CFCC7D3AC"><title>Restoring a |
|
99 previously “got” character</title> <p>This shows the use of <codeph>UnGet()</codeph> to |
|
100 restore the previously "got" character.</p> <codeblock id="GUID-8E6CACEE-5FEA-511D-B61C-9D4E99C9F1B8" xml:space="preserve">.... |
|
101 if (input.Offset() > 0) // if not at start of line |
|
102 { |
|
103 input.UnGet() ; // restore 'got' character |
|
104 }</codeblock> <p>If the previous character is before the start of the |
|
105 string, then the function raises a USER 59 panic for the <codeph>TLex8</codeph> variant |
|
106 and a USER 64 panic for the <codeph>TLex16</codeph> variant.</p> </section> |
|
107 <section id="GUID-7B2BD371-C21F-547E-8ACE-682532C0562A"><title>Reset the next |
|
108 character to the supplied mark</title> <p>This shows how to allow part of |
|
109 a <codeph>TLex</codeph> can be parsed again:</p> <codeblock id="GUID-BC84B544-599D-5A96-ACB0-7C38D1E14DBA" xml:space="preserve">if (!(input.Peek()).IsDigit()) |
|
110 { |
|
111 // found non-digit after decimal point. Error, so rewind |
|
112 input.UnGetToMark(startMark); |
|
113 }</codeblock> </section> |
|
114 <section id="GUID-D167B57E-2F96-556F-9F72-3168CC1566C6"><title>Skipping any |
|
115 non-white space to get the next token</title> <p>This parses a <codeph>TLex</codeph> for |
|
116 the next token:</p> <codeblock id="GUID-EEA10D9C-5398-54B9-A3A2-F4AC2B972AEF" xml:space="preserve">input.Mark() ; // remember where we are |
|
117 input.SkipCharacters() ; // move to end of character token |
|
118 if ( input.TokenLength() != 0 ) // if valid potential token |
|
119 ...</codeblock> </section> |
|
120 <section id="GUID-B478FB61-512C-5EEF-8230-210912F0754B"><title>Getting length |
|
121 of a token</title> <p>This shows how <codeph>TokenLength()</codeph> is used |
|
122 to return the difference between the position of the next character and the |
|
123 extraction mark. This gives a check as to whether the token length is valid. |
|
124 An invalid token length implies an invalid token.</p> <codeblock id="GUID-36BDDC14-4FCF-5D2E-B0F0-203BEFA47378" xml:space="preserve"> |
|
125 if ( input.TokenLength() != 0 ) // if valid token length |
|
126 ...</codeblock> </section> |
|
127 <section id="GUID-526E40C0-F8A5-5AC6-B71C-812BE5AAB117"><title>Extracting |
|
128 a token</title> <p>This extracts a marked token.</p> <codeblock id="GUID-448E1246-E7A1-52E7-A459-D076F9C3901C" xml:space="preserve">TPtrC token = input.MarkedToken() ; // extract token </codeblock> </section> |
|
129 <section id="GUID-37221F35-E851-5F6C-A743-A3A5C740DB8E"><title>Getting the |
|
130 offset of next character position</title> <p>This shows how to return the |
|
131 offset of the next character position from the start of the string.</p> <codeblock id="GUID-83C68224-6D43-542D-9B39-F6EF1FE43898" xml:space="preserve">if (input.Offset() > 0) // if not at start of line |
|
132 { |
|
133 input.UnGet() ; // restore 'got' character |
|
134 ... |
|
135 }</codeblock> </section> |
|
136 <section id="GUID-037C75EE-CF00-518A-92E4-89C3BD593737"><title>Extracting |
|
137 an unknown number type</title> <p>This example shows how to return the offset |
|
138 of the next character position from the start of the string.</p> <codeblock id="GUID-F510C1CB-1CF1-5816-BF83-B016ABF76429" xml:space="preserve">if (input.Val(extractUint) == KErrNone) |
|
139 { |
|
140 stack.Push(TReal(extractUint)) ; |
|
141 } |
|
142 else |
|
143 { |
|
144 if (input.Val(extractReal) == KErrNone) |
|
145 { |
|
146 stack.Push(extractReal) ; |
|
147 } |
|
148 }</codeblock> <p>This extracts an unknown number type. Tries an integer |
|
149 first and then, if this fails, tries a real:</p> <codeblock id="GUID-2A9076F2-98C9-59A8-A3C7-F03A19A5B7E8" xml:space="preserve">if (input.Val(extractUint) == KErrNone) |
|
150 stack.Push(TReal(extractUint)) ; |
|
151 else if (input.Val(extractReal) == KErrNone) |
|
152 stack.Push(extractReal) ; |
|
153 </codeblock> </section> |
|
154 </conbody></concept> |