<?xml version="1.0" encoding="utf-8"?>

<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->

<!-- This component and the accompanying materials are made available under the terms of the License 

"Eclipse Public License v1.0" which accompanies this distribution, 

and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->

<!-- Initial Contributors:

    Nokia Corporation - initial contribution.

Contributors: 

-->

<!DOCTYPE concept

  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">

<concept id="GUID-D86B83B5-298E-510A-B984-7307CCA1AE1C" xml:lang="en"><title>How to

use Lexical Analysis</title><shortdesc>Explains how to create and use TLex objects with code fragments.</shortdesc><prolog><metadata><keywords/></metadata></prolog><conbody>

<section id="GUID-8A379959-AD24-4CA7-8664-63FDDBFA2A6B"><title>Introduction</title> <p>A <codeph>TLex</codeph> object may

be constructed as an empty object, from another <codeph>TLex,</codeph> or

from an existing string as below. The code fragments are taken from an example

that implements a simple Reverse Polish Notation calculation engine.</p> <codeblock id="GUID-0F57D0AE-19C2-5C5D-BF05-5924643A933E" xml:space="preserve">TInt RPNCalc(const TDesC&amp; aCommand, TReal&amp; aReturnValue)

    {

    ...

    TLex input (aCommand) ;

    ...</codeblock> <p>Code can then proceed to move through the <codeph>TLex</codeph> data

to:</p> <ul>

<li id="GUID-DE9D6065-0426-5D33-A11B-3DEB3102A1EA"><p>mark positions for rolling

back to later</p> </li>

<li id="GUID-915C7EE1-AF85-5865-BD9A-7068426BC897"><p> delimit the start of

lexical tokens</p> </li>

<li id="GUID-0A43CFC8-B38B-5A94-AFA0-2C54266905F4"><p>delete parts of the

string held by the <codeph>TLex</codeph> object.</p> </li>

</ul> <codeblock id="GUID-2C636982-12F5-5DF6-A375-CCE4BD31FCA6" xml:space="preserve">...

input.Mark() ;                     // Remember where we are.

input.SkipCharacters() ;           // Move to end of character token.

...

_LIT(KTextMemset,"MEMSET");

if ( input.TokenLength() != 0 )    // if valid potential token

    {

    TPtrC token = input.MarkedToken() ;    // then extract token

    if ( token.CompareF(KTextMemset == 0)  // and test.

        {

        ...

        }

    ...

    }

 ...</codeblock> <p>Analysis can also be done by character using functions

that move through the <codeph>TLex</codeph> data, extracting, returning and

jumping specified character lengths. For example:</p> <codeblock id="GUID-E5650436-99E6-56C9-9CEF-2DF87F1D787D" xml:space="preserve">...

// ensure we are looking at a digit or sign

if (!(input.Peek().IsDigit() || (input.Peek() == '.') ) )

    {

    return KErrNotFound ;

    }

...    

// deal with sign

if (input.Peek() == '+')

    {

    input.Inc();

    }</codeblock> <p>Additionally, numeric conversion functions permit a variety

of numeric formats to be extracted from the <codeph>TLex</codeph> data, with

provision for conversion using the most common number systems (radixes).</p> <codeblock id="GUID-BBE6F232-7E65-58A0-BDC8-F0293E1773F7" xml:space="preserve">...

if (input.Val(extractUint) == KErrNone)

    {

    stack.Push(TReal(extractUint));

    }

else if (input.Val(extractReal) == KErrNone)

        {

        stack.Push(extractReal);

        }

...

</codeblock> <p>where <codeph>stack</codeph>, is an instance of a class implementing

a stack.</p> </section>

<section id="GUID-F688DD13-C866-5752-8530-30DECBFF0054"><title>Constructing

TLex objects</title> <p>This converts an real number into a string, which

is then assigned to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-CB88572D-66DC-5C54-9F74-02E9331A7BAD" xml:space="preserve">TBuf&lt;0x100&gt; convertRealToString;

// want a TLex from a value

if (convertRealToString.Num(value,format) < KErrNone )

    {

       ...

       }

else

    {

    convertRealToString.ZeroTerminate();

    TLex string(convertRealToString) ;

       }</codeblock> <p>This takes a descriptor as a function parameter, and

copies it to a <codeph>TLex</codeph>.</p> <codeblock id="GUID-B793E5B1-76A5-5178-B1E1-A569EA3C55EE" xml:space="preserve">TInt RPNCalc(const TDesC&amp; iCommand, TReal&amp; returnValue)

    {

    TLex input (iCommand) ;

       }</codeblock> </section>

<section id="GUID-C5B343D9-C530-5FA9-BED9-286C182081CF"><title>Peeking the

next character</title> <p>This shows a code flow decision made according to

next character to be read from the <codeph>TLex</codeph>:</p> <codeblock id="GUID-6FDB6BC3-9AB7-5C04-A003-7EE0CD26F396" xml:space="preserve">if (!(input.Peek()).IsDigit()) // found non-digit after decimal point if </codeblock> </section>

<section id="GUID-B08D3122-C580-586F-8D62-75E75A1DC717"><title>Moving past

a character that has been peeked</title> <p>This shows the use of the <codeph>Inc()</codeph> function

to move past a character that has been peeked:</p> <codeblock id="GUID-F2F1A3B4-C42E-56F0-BA5C-2F9391CCD2FB" xml:space="preserve">if (input.Peek() == '-') 

    {

    input.Inc() ;        // move past minus sign & flag

    negative = ETrue ;

    }</codeblock> </section>

<section id="GUID-BF5B871F-CAFF-50A9-8833-A79CFCC7D3AC"><title>Restoring a

previously “got” character</title> <p>This shows the use of <codeph>UnGet()</codeph> to

restore the previously "got" character.</p> <codeblock id="GUID-8E6CACEE-5FEA-511D-B61C-9D4E99C9F1B8" xml:space="preserve">....

if (input.Offset() > 0)    // if not at start of line

    {

    input.UnGet() ;        // restore 'got' character

    }</codeblock> <p>If the previous character is before the start of the

string, then the function raises a USER 59 panic for the <codeph>TLex8</codeph> variant

and a USER 64 panic for the <codeph>TLex16</codeph> variant.</p> </section>

<section id="GUID-7B2BD371-C21F-547E-8ACE-682532C0562A"><title>Reset the next

character to the supplied mark</title> <p>This shows how to allow part of

a <codeph>TLex</codeph> can be parsed again:</p> <codeblock id="GUID-BC84B544-599D-5A96-ACB0-7C38D1E14DBA" xml:space="preserve">if (!(input.Peek()).IsDigit())

    {

    // found non-digit after decimal point. Error, so rewind 

    input.UnGetToMark(startMark);

    }</codeblock> </section>

<section id="GUID-D167B57E-2F96-556F-9F72-3168CC1566C6"><title>Skipping any

non-white space to get the next token</title> <p>This parses a <codeph>TLex</codeph> for

the next token:</p> <codeblock id="GUID-EEA10D9C-5398-54B9-A3A2-F4AC2B972AEF" xml:space="preserve">input.Mark() ;                      // remember where we are

input.SkipCharacters() ;            // move to end of character token

if ( input.TokenLength() != 0 )     // if valid potential token

...</codeblock> </section>

<section id="GUID-B478FB61-512C-5EEF-8230-210912F0754B"><title>Getting length

of a token</title> <p>This shows how <codeph>TokenLength()</codeph> is used

to return the difference between the position of the next character and the

extraction mark. This gives a check as to whether the token length is valid.

An invalid token length implies an invalid token.</p> <codeblock id="GUID-36BDDC14-4FCF-5D2E-B0F0-203BEFA47378" xml:space="preserve">

if ( input.TokenLength() != 0 )  // if valid token length

...</codeblock> </section>

<section id="GUID-526E40C0-F8A5-5AC6-B71C-812BE5AAB117"><title>Extracting

a token</title> <p>This extracts a marked token.</p> <codeblock id="GUID-448E1246-E7A1-52E7-A459-D076F9C3901C" xml:space="preserve">TPtrC token = input.MarkedToken() ;  // extract token </codeblock> </section>

<section id="GUID-37221F35-E851-5F6C-A743-A3A5C740DB8E"><title>Getting the

offset of next character position</title> <p>This shows how to return the

offset of the next character position from the start of the string.</p> <codeblock id="GUID-83C68224-6D43-542D-9B39-F6EF1FE43898" xml:space="preserve">if (input.Offset() &gt; 0)    // if not at start of line

    {

    input.UnGet() ;        // restore 'got' character

    ...

    }</codeblock> </section>

<section id="GUID-037C75EE-CF00-518A-92E4-89C3BD593737"><title>Extracting

an unknown number type</title> <p>This example shows how to return the offset

of the next character position from the start of the string.</p> <codeblock id="GUID-F510C1CB-1CF1-5816-BF83-B016ABF76429" xml:space="preserve">if (input.Val(extractUint) == KErrNone)

    {

    stack.Push(TReal(extractUint)) ;

    }

else 

    {

    if (input.Val(extractReal) == KErrNone)

        {

        stack.Push(extractReal) ;

        }

    }</codeblock> <p>This extracts an unknown number type. Tries an integer

first and then, if this fails, tries a real:</p> <codeblock id="GUID-2A9076F2-98C9-59A8-A3C7-F03A19A5B7E8" xml:space="preserve">if (input.Val(extractUint) == KErrNone)

    stack.Push(TReal(extractUint)) ;

else if (input.Val(extractReal) == KErrNone)

    stack.Push(extractReal) ;

</codeblock> </section>