1 <?xml version="1.0" encoding="utf-8"?>

     2 <!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->

     3 <!-- This component and the accompanying materials are made available under the terms of the License 

     4 "Eclipse Public License v1.0" which accompanies this distribution, 

     5 and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->

     6 <!-- Initial Contributors:

     7     Nokia Corporation - initial contribution.

     8 Contributors: 

     9 -->

    10 <!DOCTYPE concept

    11   PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">

    12 <concept id="GUID-2E22CB5B-E648-5760-AB74-8947C7EDE2CF" xml:lang="en"><title>Lexical

    13 Analysis Overview</title><shortdesc>Provides string analysis and string-to-number conversions</shortdesc><prolog><metadata><keywords/></metadata></prolog><conbody>

    14 <section id="GUID-A64D2A5B-2A8C-4FEF-9F4A-6AC4F997BB7F"><title>Description</title> <p>The API has two key concepts: lexical

    15 analyser, and extraction mark. </p> </section>

    16 <section id="GUID-12BA42C0-484F-46FF-A9EB-8F8968B2F272"><title>Lexical analyser</title><p>The lexical analyser provides facilities

    17 to identify and extract characters and tokens (substrings delimited by white

    18 space) from a descriptor or NULL-terminated string. It also provides string-to-number

    19 conversions.</p> <p>The lexical analyser interface is provided by <xref href="GUID-76BFE569-34F7-3FEC-8F01-38AEC8C29D68.dita"><apiname>TLex16</apiname></xref> for

    20 wide strings, and <xref href="GUID-5B7955BC-1375-35F1-953D-D7F5C3583756.dita"><apiname>TLex8</apiname></xref> for narrow strings. <xref href="GUID-C94AB4F0-AE42-3957-A037-77CB145DDBF8.dita"><apiname>TLex</apiname></xref> is

    21 a typedef for <xref href="GUID-76BFE569-34F7-3FEC-8F01-38AEC8C29D68.dita"><apiname>TLex16</apiname></xref>.</p></section>

    22 <section id="GUID-A1CF4E86-0B08-4D16-A352-2240983CD651"><title>Extraction mark</title><p>The extraction mark allows positions

    23 in a string that is being analysed to be remembered. Later lexical analysis

    24 operations can then operate on this position in the string.</p> <p>The extraction

    25 mark interface is provided by <xref href="GUID-284AF88B-910B-3483-8797-E85ECB08BB2A.dita"><apiname>TLexMark16</apiname></xref> for wide strings,

    26 and <xref href="GUID-11EFB436-59E4-3E21-8950-B7F3F16FDD92.dita"><apiname>TLexMark8</apiname></xref> for narrow strings. <xref href="GUID-CCFA0F79-F758-3728-AA4E-DDF59D3817C1.dita"><apiname>TLexMark</apiname></xref> is

    27 a typedef for <xref href="GUID-284AF88B-910B-3483-8797-E85ECB08BB2A.dita"><apiname>TLexMark16</apiname></xref>.</p></section>

    28 </conbody><related-links>

    29 <link>

    30 <desc><xref href="GUID-0817AD1D-58CF-5108-ACBF-26DFD4BA395E.dita">Descriptors Overview</xref></desc>

    31 </link>

    32 </related-links></concept>