<?xml version="1.0" encoding="utf-8"?>
<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->
<!-- This component and the accompanying materials are made available under the terms of the License 
"Eclipse Public License v1.0" which accompanies this distribution, 
and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->
<!-- Initial Contributors:
    Nokia Corporation - initial contribution.
Contributors: 
-->
<!DOCTYPE concept
  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="GUID-D86B83B5-298E-510A-B984-7307CCA1AE1C" xml:lang="en"><title>How to
use Lexical Analysis</title><shortdesc>Explains how to create and use TLex objects with code fragments.</shortdesc><prolog><metadata><keywords/></metadata></prolog><conbody>
<section id="GUID-8A379959-AD24-4CA7-8664-63FDDBFA2A6B"><title>Introduction</title> <p>A <codeph>TLex</codeph> object may
be constructed as an empty object, from another <codeph>TLex,</codeph> or
from an existing string as below. The code fragments are taken from an example
that implements a simple Reverse Polish Notation calculation engine.</p> <codeblock id="GUID-0F57D0AE-19C2-5C5D-BF05-5924643A933E" xml:space="preserve">TInt RPNCalc(const TDesC&amp; aCommand, TReal&amp; aReturnValue)
    {
    ...
    TLex input (aCommand) ;
    ...</codeblock> <p>Code can then proceed to move through the <codeph>TLex</codeph> data
to:</p> <ul>
<li id="GUID-DE9D6065-0426-5D33-A11B-3DEB3102A1EA"><p>mark positions for rolling
back to later</p> </li>
<li id="GUID-915C7EE1-AF85-5865-BD9A-7068426BC897"><p> delimit the start of
lexical tokens</p> </li>
<li id="GUID-0A43CFC8-B38B-5A94-AFA0-2C54266905F4"><p>delete parts of the
string held by the <codeph>TLex</codeph> object.</p> </li>
</ul> <codeblock id="GUID-2C636982-12F5-5DF6-A375-CCE4BD31FCA6" xml:space="preserve">...
input.Mark() ;                     // Remember where we are.
input.SkipCharacters() ;           // Move to end of character token.
...
_LIT(KTextMemset,"MEMSET");
if ( input.TokenLength() != 0 )    // if valid potential token
    {
    TPtrC token = input.MarkedToken() ;    // then extract token
    if ( token.CompareF(KTextMemset == 0)  // and test.
        {
        ...
        }
    ...
    }
 ...</codeblock> <p>Analysis can also be done by character using functions
that move through the <codeph>TLex</codeph> data, extracting, returning and
jumping specified character lengths. For example:</p> <codeblock id="GUID-E5650436-99E6-56C9-9CEF-2DF87F1D787D" xml:space="preserve">...
// ensure we are looking at a digit or sign
if (!(input.Peek().IsDigit() || (input.Peek() == '.') ) )
    {
    return KErrNotFound ;
    }
...    
// deal with sign
if (input.Peek() == '+')
    {
    input.Inc();
    }</codeblock> <p>Additionally, numeric conversion functions permit a variety
of numeric formats to be extracted from the <codeph>TLex</codeph> data, with
provision for conversion using the most common number systems (radixes).</p> <codeblock id="GUID-BBE6F232-7E65-58A0-BDC8-F0293E1773F7" xml:space="preserve">...
if (input.Val(extractUint) == KErrNone)
    {
    stack.Push(TReal(extractUint));
    }
else if (input.Val(extractReal) == KErrNone)
        {
        stack.Push(extractReal);
        }
...

</codeblock> <p>where <codeph>stack</codeph>, is an instance of a class implementing
a stack.</p> </section>
<section id="GUID-F688DD13-C866-5752-8530-30DECBFF0054"><title>Constructing
TLex objects</title> <p>This converts an real number into a string, which
is then assigned to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-CB88572D-66DC-5C54-9F74-02E9331A7BAD" xml:space="preserve">TBuf&lt;0x100&gt; convertRealToString;
// want a TLex from a value
if (convertRealToString.Num(value,format) &lt; KErrNone )
    {
       ...
       }
else
    {
    convertRealToString.ZeroTerminate();
    TLex string(convertRealToString) ;
       }</codeblock> <p>This takes a descriptor as a function parameter, and
copies it to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-B793E5B1-76A5-5178-B1E1-A569EA3C55EE" xml:space="preserve">TInt RPNCalc(const TDesC&amp; iCommand, TReal&amp; returnValue)
    {
    TLex input (iCommand) ;
       }</codeblock> </section>
<section id="GUID-C5B343D9-C530-5FA9-BED9-286C182081CF"><title>Peeking the
next character</title> <p>This shows a code flow decision made according to
next character to be read from the <codeph>TLex</codeph>:</p> <codeblock id="GUID-6FDB6BC3-9AB7-5C04-A003-7EE0CD26F396" xml:space="preserve">if (!(input.Peek()).IsDigit()) // found non-digit after decimal point if </codeblock> </section>
<section id="GUID-B08D3122-C580-586F-8D62-75E75A1DC717"><title>Moving past
a character that has been peeked</title> <p>This shows the use of the <codeph>Inc()</codeph> function
to move past a character that has been peeked:</p> <codeblock id="GUID-F2F1A3B4-C42E-56F0-BA5C-2F9391CCD2FB" xml:space="preserve">if (input.Peek() == '-') 
    {
    input.Inc() ;        // move past minus sign &amp; flag
    negative = ETrue ;
    }</codeblock> </section>
<section id="GUID-BF5B871F-CAFF-50A9-8833-A79CFCC7D3AC"><title>Restoring a
previously “got” character</title> <p>This shows the use of <codeph>UnGet()</codeph> to
restore the previously "got" character.</p> <codeblock id="GUID-8E6CACEE-5FEA-511D-B61C-9D4E99C9F1B8" xml:space="preserve">....
if (input.Offset() &gt; 0)    // if not at start of line
    {
    input.UnGet() ;        // restore 'got' character
    }</codeblock> <p>If the previous character is before the start of the
string, then the function raises a USER 59 panic for the <codeph>TLex8</codeph> variant
and a USER 64 panic for the <codeph>TLex16</codeph> variant.</p> </section>
<section id="GUID-7B2BD371-C21F-547E-8ACE-682532C0562A"><title>Reset the next
character to the supplied mark</title> <p>This shows how to allow part of
a <codeph>TLex</codeph> can be parsed again:</p> <codeblock id="GUID-BC84B544-599D-5A96-ACB0-7C38D1E14DBA" xml:space="preserve">if (!(input.Peek()).IsDigit())
    {
    // found non-digit after decimal point. Error, so rewind 
    input.UnGetToMark(startMark);
    }</codeblock> </section>
<section id="GUID-D167B57E-2F96-556F-9F72-3168CC1566C6"><title>Skipping any
non-white space to get the next token</title> <p>This parses a <codeph>TLex</codeph> for
the next token:</p> <codeblock id="GUID-EEA10D9C-5398-54B9-A3A2-F4AC2B972AEF" xml:space="preserve">input.Mark() ;                      // remember where we are
input.SkipCharacters() ;            // move to end of character token
if ( input.TokenLength() != 0 )     // if valid potential token
...</codeblock> </section>
<section id="GUID-B478FB61-512C-5EEF-8230-210912F0754B"><title>Getting length
of a token</title> <p>This shows how <codeph>TokenLength()</codeph> is used
to return the difference between the position of the next character and the
extraction mark. This gives a check as to whether the token length is valid.
An invalid token length implies an invalid token.</p> <codeblock id="GUID-36BDDC14-4FCF-5D2E-B0F0-203BEFA47378" xml:space="preserve">
if ( input.TokenLength() != 0 )  // if valid token length
...</codeblock> </section>
<section id="GUID-526E40C0-F8A5-5AC6-B71C-812BE5AAB117"><title>Extracting
a token</title> <p>This extracts a marked token.</p> <codeblock id="GUID-448E1246-E7A1-52E7-A459-D076F9C3901C" xml:space="preserve">TPtrC token = input.MarkedToken() ;  // extract token </codeblock> </section>
<section id="GUID-37221F35-E851-5F6C-A743-A3A5C740DB8E"><title>Getting the
offset of next character position</title> <p>This shows how to return the
offset of the next character position from the start of the string.</p> <codeblock id="GUID-83C68224-6D43-542D-9B39-F6EF1FE43898" xml:space="preserve">if (input.Offset() &gt; 0)    // if not at start of line
    {
    input.UnGet() ;        // restore 'got' character
    ...
    }</codeblock> </section>
<section id="GUID-037C75EE-CF00-518A-92E4-89C3BD593737"><title>Extracting
an unknown number type</title> <p>This example shows how to return the offset
of the next character position from the start of the string.</p> <codeblock id="GUID-F510C1CB-1CF1-5816-BF83-B016ABF76429" xml:space="preserve">if (input.Val(extractUint) == KErrNone)
    {
    stack.Push(TReal(extractUint)) ;
    }
else 
    {
    if (input.Val(extractReal) == KErrNone)
        {
        stack.Push(extractReal) ;
        }
    }</codeblock> <p>This extracts an unknown number type. Tries an integer
first and then, if this fails, tries a real:</p> <codeblock id="GUID-2A9076F2-98C9-59A8-A3C7-F03A19A5B7E8" xml:space="preserve">if (input.Val(extractUint) == KErrNone)
    stack.Push(TReal(extractUint)) ;
else if (input.Val(extractReal) == KErrNone)
    stack.Push(extractReal) ;
</codeblock> </section>
</conbody></concept>