<?xml version="1.0" encoding="utf-8"?>

<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->

<!-- This component and the accompanying materials are made available under the terms of the License 

"Eclipse Public License v1.0" which accompanies this distribution, 

and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->

<!-- Initial Contributors:

    Nokia Corporation - initial contribution.

Contributors: 

-->

<!DOCTYPE concept

  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">

<concept id="GUID-C501E703-E39D-598C-B962-7A32AC9091DD" xml:lang="en"><title>Folding

and collation (comparing strings)</title><shortdesc>Describes descriptor folding and descriptor collation.</shortdesc><prolog><metadata><keywords/></metadata></prolog><conbody>

<p>There are two techniques that may be used to modify the characters in a

descriptor prior to performing operations such as comparisons on text strings: </p>

<ul>

<li id="GUID-C550B19E-0312-52F4-936C-95C53D4D4FA9"><p>folding </p> </li>

<li id="GUID-77D0249A-F9B7-5189-808F-C4FB88BED5B3"><p>collation </p> </li>

</ul>

<section id="GUID-4AD769A8-A90B-4BE5-B514-DCE9C808C4A8"><title>Folding</title> <p>Folding is a relatively simple way of normalising

text for comparison by removing case distinctions, converting accented characters

to characters without accents etc. Folding is used for tolerant comparisons,

i.e. comparisons that are biased towards a match. </p> <p>For example, the

file system uses folding to decide whether two file names are identical or

not. Folding is locale-independent behaviour, and means that the file system,

for example, can be locale-independent. </p> <p> <i> It is important to note

that there can be no guarantee that folding is in any way culturally appropriate,

and should not be used for comparing strings in natural language; </i> <xref href="GUID-C501E703-E39D-598C-B962-7A32AC9091DD.dita#GUID-C501E703-E39D-598C-B962-7A32AC9091DD/GUID-F93D3C40-FDB4-5D92-A90C-736BB0225982">collation</xref> <i>is

the correct functionality for this.</i>  </p> <p>Variants of member functions

that fold are provided where appropriate. For example, <xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-7BDF7FA1-39FF-35D2-97DE-12A223514345"><apiname>TDesC16::CompareF()</apiname></xref> for

folded comparison. </p> <p>See also: </p><p><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-7BDF7FA1-39FF-35D2-97DE-12A223514345"><apiname>TDesC16::CompareF()</apiname></xref><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-57DED784-A51D-308B-888C-968EFB35B732"><apiname>TDesC16::MatchF()</apiname></xref><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-D4BDA3FC-E11A-392B-A8A5-B468AC800396"><apiname>TDesC16::FindF()</apiname></xref><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-F88740FB-C90A-30AF-AA19-E2260EB39A47"><apiname>TDesC16::LocateF()</apiname></xref> <xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-F88740FB-C90A-30AF-AA19-E2260EB39A47"><apiname>TDesC16::LocateF()</apiname></xref> <xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-BE28DE82-AEF1-3E71-A0E1-7A053095B5B0"><apiname>TDesC16::LocateReverseF()</apiname></xref></p> </section>

<section id="GUID-F93D3C40-FDB4-5D92-A90C-736BB0225982"><title>Collation</title> <p>Collation

is a much better and more powerful way to compare strings and produces a dictionary-like

('lexicographic') ordering. Folding cannot remove piece accents or deal with

correspondences that are not one-to-one like the mapping from German upper

case SS to lower case ß. In addition, folding cannot optionally ignore punctuation. </p> <p>For

languages using the Latin script, for example, collation is about deciding

whether to ignore punctuation, whether to fold upper and lower case, how to

treat accents, and so on. In a given locale there is usually a standard set

of collation rules that can be used. </p> <p> <i>Collation should always be

used for comparing strings in natural language.</i>  </p> <p>Variants of member

functions that use collation are provided where appropriate. For example, <xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-8B44C890-6E64-37CF-B3D9-AEF9EFCBA284"><apiname>TDesC16::CompareC()</apiname></xref> for

collated comparison. </p> <p><b>Comparing

and sorting strings</b> </p> <p>The <xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-8B44C890-6E64-37CF-B3D9-AEF9EFCBA284"><apiname>TDesC16::CompareC()</apiname></xref> variant

prototyped as: </p> <codeblock id="GUID-42E8C509-DA19-50AD-8A80-D381F812639A" xml:space="preserve">TInt CompareC(const TDesC16&amp; aDes, TInt aMaxLevel, const TCollationMethod* aCollationMethod) const;</codeblock> <p>returns 0, if two strings match. </p> <p>There are many ways in which

two strings can match, even when they do not have the same length: </p> <ul>

<li id="GUID-8BD9DEF8-3B24-576D-B634-0E38BA2D5859"><p>if one string includes

combining characters, but the collation level is set to 0 (which means that

accents are ignored) </p> </li>

<li id="GUID-999CCABF-D8DC-5967-90F1-7739EDCAEFF9"><p>if one string contains

"pre-composed" versions of accented characters and the other contains "decomposed"

versions of the same character </p> </li>

<li id="GUID-9491E1CF-9BA3-52D6-8996-FC32BB54538A"><p>if one string contains

a ligature that, in a collation table, matches multiple characters in the

other string and the collation level is set to less than 3 (for example "æ"

might match "ae") </p> </li>

<li id="GUID-2D4FF198-7C41-5838-97FB-8A23A2D3DD49"><p>if one string contains

a "surrogate pair" (a 32-bit encoded character) which happens to match a normal

character at the level specified </p> </li>

<li id="GUID-66B770C7-3215-559A-B70D-68FF3AC9DDAC"><p>if the collation method

does not have its "ignore none" flag set and the collation level is set to

less than 3, then spaces and punctuation are ignored; this means that one

string could be much longer than the other just by adding a large number of

spaces </p> </li>

<li id="GUID-33026C64-3740-527A-97E0-DCBB6D7F087D"><p>if one string were to

contain the Hangul representation of Korean and the other were to contain

the Jamo representation of the same Korean and the collation level is set

to less than 3. </p> </li>

</ul> <p>The collation level is an integer that can take one of the values:

0, 1, 2 or 3, and determines how tightly the matching of two strings should

be. This value is passed as the second parameter to <codeph>CompareC()</codeph>.

The values have the following meanings: </p> <ul>

<li id="GUID-EDA2DAB4-E6B4-5815-ADC1-1BDF216D0C2E"><p>0 - only test the character

identity; accents and case are ignored </p> </li>

<li id="GUID-B0300A5F-6AF5-5722-B538-E0DBDE25576B"><p>1 - test the character

identity and accents; case is ignored </p> </li>

<li id="GUID-B6B7E302-120D-5240-B2AE-C14B19A36FF6"><p>2 - test the character

identity, accents and case </p> </li>

<li id="GUID-22E919E8-819D-5DE2-BABA-ECCC6F105B3C"><p>3 - test the Unicode

value as well as the character identity, accents and case. </p> </li>

</ul> <p>At levels 0-2: </p> <ul>

<li id="GUID-E08F7C36-2E53-5348-A8D4-6A1E420CEB76"><p>ligatures (e.g. "æ")

are the same as their decomposed equivalents (e.g. "ae") </p> </li>

<li id="GUID-937B9898-878A-59BE-B185-A5347038F14C"><p>script variants are

the same (for example "R" matches the mathematical real number symbol (Unicode

211D) </p> </li>

<li id="GUID-6D1AF8EB-E8E8-565E-BA1E-CAED0382D801"><p>the "micro" symbol (Unicode

00B5) matches Greek "mu" (Unicode 03BC)). </p> </li>

</ul> <p>At level 3 these are treated differently. </p> <p>If the aim is to <b>sort</b> strings,

then <b>level 3 must be used</b>. For any strings <codeph>a</codeph> and <codeph>b</codeph>,

if <codeph>a</codeph> &lt; <codeph>b</codeph> for some level of collation,

then <codeph>              a</codeph> &lt; <codeph>b</codeph> for all higher

levels of collation as well. It is impossible, therefore, to affect the order

that is generated by using lower collation levels than 3. This just causes

similar strings to sort in a random order. In standard English, sorting at

level 3 gives the following order: </p> <p>bat &lt; bee &lt; BEE &lt; bus </p> <p>The

case of the B only affects the comparison after all the letter identities

have been found to be the same - this is usually what people are trying to

achieve by using lower collation levels than 3 for sorting. It is never necessary. </p> <p>The

sort order can be affected by setting flags in the <xref href="GUID-78C4965C-BFCD-3E7E-8F46-2EE3D1BAF6EC.dita"><apiname>TCollationMethod</apiname></xref> object. </p> <p>Note

that when strings match at level 3, they do not necessarily have the same

binary representation, or even the same length. Unicode contains many strings

that are regarded as equivalent, even though they have different binary representations. </p><p> 

       See also:  </p><p><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-8B44C890-6E64-37CF-B3D9-AEF9EFCBA284"><apiname>TDesC16::CompareC()</apiname></xref><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-ACEEA02F-2594-3C61-B7A9-E96F0737C3AE"><apiname>TDesC16::MatchC()</apiname></xref><xref href="GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23.dita#GUID-440FF2B4-353B-3097-A2BA-5887D10B8B23/GUID-33D33034-0757-31F9-B3A2-BA351AADC816"><apiname>TDesC16::FindC()</apiname></xref></p> </section>