--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Symbian3/PDK/Source/GUID-D86B83B5-298E-510A-B984-7307CCA1AE1C.dita	Fri Jan 22 18:26:19 2010 +0000
@@ -0,0 +1,154 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->
+<!-- This component and the accompanying materials are made available under the terms of the License 
+"Eclipse Public License v1.0" which accompanies this distribution, 
+and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->
+<!-- Initial Contributors:
+    Nokia Corporation - initial contribution.
+Contributors: 
+-->
+<!DOCTYPE concept
+  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="GUID-D86B83B5-298E-510A-B984-7307CCA1AE1C" xml:lang="en"><title>How to
+use Lexical Analysis</title><shortdesc>Explains how to create and use TLex objects with code fragments.</shortdesc><prolog><metadata><keywords/></metadata></prolog><conbody>
+<section id="GUID-8A379959-AD24-4CA7-8664-63FDDBFA2A6B"><title>Introduction</title> <p>A <codeph>TLex</codeph> object may
+be constructed as an empty object, from another <codeph>TLex,</codeph> or
+from an existing string as below. The code fragments are taken from an example
+that implements a simple Reverse Polish Notation calculation engine.</p> <codeblock id="GUID-0F57D0AE-19C2-5C5D-BF05-5924643A933E" xml:space="preserve">TInt RPNCalc(const TDesC&amp; aCommand, TReal&amp; aReturnValue)
+    {
+    ...
+    TLex input (aCommand) ;
+    ...</codeblock> <p>Code can then proceed to move through the <codeph>TLex</codeph> data
+to:</p> <ul>
+<li id="GUID-DE9D6065-0426-5D33-A11B-3DEB3102A1EA"><p>mark positions for rolling
+back to later</p> </li>
+<li id="GUID-915C7EE1-AF85-5865-BD9A-7068426BC897"><p> delimit the start of
+lexical tokens</p> </li>
+<li id="GUID-0A43CFC8-B38B-5A94-AFA0-2C54266905F4"><p>delete parts of the
+string held by the <codeph>TLex</codeph> object.</p> </li>
+</ul> <codeblock id="GUID-2C636982-12F5-5DF6-A375-CCE4BD31FCA6" xml:space="preserve">...
+input.Mark() ;                     // Remember where we are.
+input.SkipCharacters() ;           // Move to end of character token.
+...
+_LIT(KTextMemset,"MEMSET");
+if ( input.TokenLength() != 0 )    // if valid potential token
+    {
+    TPtrC token = input.MarkedToken() ;    // then extract token
+    if ( token.CompareF(KTextMemset == 0)  // and test.
+        {
+        ...
+        }
+    ...
+    }
+ ...</codeblock> <p>Analysis can also be done by character using functions
+that move through the <codeph>TLex</codeph> data, extracting, returning and
+jumping specified character lengths. For example:</p> <codeblock id="GUID-E5650436-99E6-56C9-9CEF-2DF87F1D787D" xml:space="preserve">...
+// ensure we are looking at a digit or sign
+if (!(input.Peek().IsDigit() || (input.Peek() == '.') ) )
+    {
+    return KErrNotFound ;
+    }
+...    
+// deal with sign
+if (input.Peek() == '+')
+    {
+    input.Inc();
+    }</codeblock> <p>Additionally, numeric conversion functions permit a variety
+of numeric formats to be extracted from the <codeph>TLex</codeph> data, with
+provision for conversion using the most common number systems (radixes).</p> <codeblock id="GUID-BBE6F232-7E65-58A0-BDC8-F0293E1773F7" xml:space="preserve">...
+if (input.Val(extractUint) == KErrNone)
+    {
+    stack.Push(TReal(extractUint));
+    }
+else if (input.Val(extractReal) == KErrNone)
+        {
+        stack.Push(extractReal);
+        }
+...
+
+</codeblock> <p>where <codeph>stack</codeph>, is an instance of a class implementing
+a stack.</p> </section>
+<section id="GUID-F688DD13-C866-5752-8530-30DECBFF0054"><title>Constructing
+TLex objects</title> <p>This converts an real number into a string, which
+is then assigned to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-CB88572D-66DC-5C54-9F74-02E9331A7BAD" xml:space="preserve">TBuf&lt;0x100&gt; convertRealToString;
+// want a TLex from a value
+if (convertRealToString.Num(value,format) &lt; KErrNone )
+    {
+       ...
+       }
+else
+    {
+    convertRealToString.ZeroTerminate();
+    TLex string(convertRealToString) ;
+       }</codeblock> <p>This takes a descriptor as a function parameter, and
+copies it to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-B793E5B1-76A5-5178-B1E1-A569EA3C55EE" xml:space="preserve">TInt RPNCalc(const TDesC&amp; iCommand, TReal&amp; returnValue)
+    {
+    TLex input (iCommand) ;
+       }</codeblock> </section>
+<section id="GUID-C5B343D9-C530-5FA9-BED9-286C182081CF"><title>Peeking the
+next character</title> <p>This shows a code flow decision made according to
+next character to be read from the <codeph>TLex</codeph>:</p> <codeblock id="GUID-6FDB6BC3-9AB7-5C04-A003-7EE0CD26F396" xml:space="preserve">if (!(input.Peek()).IsDigit()) // found non-digit after decimal point if </codeblock> </section>
+<section id="GUID-B08D3122-C580-586F-8D62-75E75A1DC717"><title>Moving past
+a character that has been peeked</title> <p>This shows the use of the <codeph>Inc()</codeph> function
+to move past a character that has been peeked:</p> <codeblock id="GUID-F2F1A3B4-C42E-56F0-BA5C-2F9391CCD2FB" xml:space="preserve">if (input.Peek() == '-') 
+    {
+    input.Inc() ;        // move past minus sign &amp; flag
+    negative = ETrue ;
+    }</codeblock> </section>
+<section id="GUID-BF5B871F-CAFF-50A9-8833-A79CFCC7D3AC"><title>Restoring a
+previously “got” character</title> <p>This shows the use of <codeph>UnGet()</codeph> to
+restore the previously "got" character.</p> <codeblock id="GUID-8E6CACEE-5FEA-511D-B61C-9D4E99C9F1B8" xml:space="preserve">....
+if (input.Offset() &gt; 0)    // if not at start of line
+    {
+    input.UnGet() ;        // restore 'got' character
+    }</codeblock> <p>If the previous character is before the start of the
+string, then the function raises a USER 59 panic for the <codeph>TLex8</codeph> variant
+and a USER 64 panic for the <codeph>TLex16</codeph> variant.</p> </section>
+<section id="GUID-7B2BD371-C21F-547E-8ACE-682532C0562A"><title>Reset the next
+character to the supplied mark</title> <p>This shows how to allow part of
+a <codeph>TLex</codeph> can be parsed again:</p> <codeblock id="GUID-BC84B544-599D-5A96-ACB0-7C38D1E14DBA" xml:space="preserve">if (!(input.Peek()).IsDigit())
+    {
+    // found non-digit after decimal point. Error, so rewind 
+    input.UnGetToMark(startMark);
+    }</codeblock> </section>
+<section id="GUID-D167B57E-2F96-556F-9F72-3168CC1566C6"><title>Skipping any
+non-white space to get the next token</title> <p>This parses a <codeph>TLex</codeph> for
+the next token:</p> <codeblock id="GUID-EEA10D9C-5398-54B9-A3A2-F4AC2B972AEF" xml:space="preserve">input.Mark() ;                      // remember where we are
+input.SkipCharacters() ;            // move to end of character token
+if ( input.TokenLength() != 0 )     // if valid potential token
+...</codeblock> </section>
+<section id="GUID-B478FB61-512C-5EEF-8230-210912F0754B"><title>Getting length
+of a token</title> <p>This shows how <codeph>TokenLength()</codeph> is used
+to return the difference between the position of the next character and the
+extraction mark. This gives a check as to whether the token length is valid.
+An invalid token length implies an invalid token.</p> <codeblock id="GUID-36BDDC14-4FCF-5D2E-B0F0-203BEFA47378" xml:space="preserve">
+if ( input.TokenLength() != 0 )  // if valid token length
+...</codeblock> </section>
+<section id="GUID-526E40C0-F8A5-5AC6-B71C-812BE5AAB117"><title>Extracting
+a token</title> <p>This extracts a marked token.</p> <codeblock id="GUID-448E1246-E7A1-52E7-A459-D076F9C3901C" xml:space="preserve">TPtrC token = input.MarkedToken() ;  // extract token </codeblock> </section>
+<section id="GUID-37221F35-E851-5F6C-A743-A3A5C740DB8E"><title>Getting the
+offset of next character position</title> <p>This shows how to return the
+offset of the next character position from the start of the string.</p> <codeblock id="GUID-83C68224-6D43-542D-9B39-F6EF1FE43898" xml:space="preserve">if (input.Offset() &gt; 0)    // if not at start of line
+    {
+    input.UnGet() ;        // restore 'got' character
+    ...
+    }</codeblock> </section>
+<section id="GUID-037C75EE-CF00-518A-92E4-89C3BD593737"><title>Extracting
+an unknown number type</title> <p>This example shows how to return the offset
+of the next character position from the start of the string.</p> <codeblock id="GUID-F510C1CB-1CF1-5816-BF83-B016ABF76429" xml:space="preserve">if (input.Val(extractUint) == KErrNone)
+    {
+    stack.Push(TReal(extractUint)) ;
+    }
+else 
+    {
+    if (input.Val(extractReal) == KErrNone)
+        {
+        stack.Push(extractReal) ;
+        }
+    }</codeblock> <p>This extracts an unknown number type. Tries an integer
+first and then, if this fails, tries a real:</p> <codeblock id="GUID-2A9076F2-98C9-59A8-A3C7-F03A19A5B7E8" xml:space="preserve">if (input.Val(extractUint) == KErrNone)
+    stack.Push(TReal(extractUint)) ;
+else if (input.Val(extractReal) == KErrNone)
+    stack.Push(extractReal) ;
+</codeblock> </section>
+</conbody></concept>
\ No newline at end of file