<?xml version="1.0" encoding="utf-8"?>

<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->

<!-- This component and the accompanying materials are made available under the terms of the License 

"Eclipse Public License v1.0" which accompanies this distribution, 

and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->

<!-- Initial Contributors:

    Nokia Corporation - initial contribution.

Contributors: 

-->

<!DOCTYPE concept

  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">

<concept id="GUID-06EDE5E8-04EA-5A74-ADE2-E5B8C49AB292" xml:lang="en"><title>Character

Conversion (Charconv) Framework Concepts</title><prolog><metadata><keywords/></metadata></prolog><conbody>

<p>This section describes the terminology used often in character conversions,

such as BMP and Charconv converters. </p>

<section id="GUID-F964FD3C-D80B-4DBB-A99D-71CC60C362FC"><title>Character sets </title> <p>Textual data in electronic devices

is stored in terms of a character set. A character set is a group of characters,

each of which is encoded as a different number. The appearance of each character

is not a property of the character set, but rather of the font. So a character

may be rendered using many different glyphs, but will always have the same

numeric value within its character set. Other properties which can also be

included in a character set’s definition are the direction of writing, and

the way in which sets of characters are combined. </p> </section>

<section id="GUID-58021C48-1A3D-41C8-8B82-16C0481BFDCB"><title>Unicode, UCS and UCS-2</title> <p>Character sets, and the

ways of encoding them, have proliferated with the increasing acceptance of

computers and communicators throughout the world. This has led to an international

standard character set, which encompasses all commonly used character sets,

including Eastern ideograms, in a single character set, Unicode, defined by

the Unicode Consortium (http://www.unicode.org). </p> <p>UCS is the name for

Unicode Character Set. Unicode characters are generally encoded using one

16-bit value but written to files in two bytes. This is referred to as UCS-2

encoding formats. There are also other Unicode encoding formats such as UTF-16

and UTF-8 for different purposes. For the full definition of these formats,

see The Unicode Standard published by the Unicode Consortium. </p> </section>

<section id="GUID-24F61FEA-C3FE-4CBB-BDA2-4FF741288B63"><title>BMP</title> <p>Unicode points between U+0000 to U+FFFF are

called Basic Multilingual Plane (BMP). BMP covers almost all characters in

different languages. Code points outside the BMP must be encoded using a "surrogate

pair", which consists of two 16-bit values. The Symbian platform

currently does not support scripts with characters mapped to code points above

U+FFFF. Code points above U+FFFF are also known as supplementary characters. </p> </section>

<section id="GUID-21DF5FEF-2446-4D23-8139-869A0CD7B514"><title> UTF-16</title> <p>UTF-16 is one of the Unicode encoding formats.

It supports characters within and outside BMP using a number of 16-bit characters. </p> <p>In

the text-processing subsystem, the Symbian platform uses UTF-16 Unicode format.

This means that any input to the text-processing subsystem must be in UTF-16.

Different character converters can be used to convert text from other encoding

formats to UTF-16. </p> </section>

<section id="GUID-786FEE95-D7A5-4E41-AB41-C8D54BFB8C54"><title>Transformation formats </title> <p>The UCS-2 format of the

Unicode character set encodes each character as 2 bytes (16 bits total). However

it does not specify which of the bytes is most significant. The byte order,

or endian-ness, is left up to the discretion of a particular operating system. </p> <p>While

this is not important within a system, it does mean that text encoded as UCS-2

cannot easily be shared between systems using a different endian-ness. To

overcome this problem the Unicode Consortium has defined two transformation

formats for sharing Unicode text. The transformation formats explicitly specify

byte order, and cannot be misinterpreted by computers using a different byte

order. </p> <p>The two transformation formats, UTF-7 and UTF-8, are described

below. For the full definition of these formats, see The Unicode Standard

published by the Unicode Consortium. </p> <p> <b>UTF-7</b>  </p> <p>UTF-7

allows Unicode characters to be encoded and transmitted as 8-bit bytes, of

which only 7 bits are used. UTF-7 divides the set of Unicode characters into

three subsets, which are encoded and transmitted differently. </p> <ul>

<li id="GUID-8E1A1C8B-8234-57C3-93D4-5A0A4E8C1374"><p>Set D, is the set of

characters which are encoded as a single byte. It includes lower and upper

case A to Z, the numeric digits, and nine other characters. </p> </li>

<li id="GUID-3E19560B-4087-575E-A091-64FCFD24C811"><p>Set O includes the characters <b>!

" # $ % &amp; * ; &lt; = &gt; @ [ ] ^ _ </b> <b>{</b> <b> | </b> <b>}</b>. These

characters can be encoded as a single byte, or with the modified <keyword>base

                64</keyword>  encoding used for set B characters. When

encoded as a single byte, set O characters can be misinterpreted by some applications —

encoding as modified base 64 overcomes this problem. </p> </li>

<li id="GUID-01E822AE-71CC-5F0B-BC60-53F914600A5E"><p>Set B comprises the

remaining characters, which are encoded as an <keyword>escape byte</keyword> followed

by 2 or 3 bytes. The encoding format is a modified form of base 64 encoding. </p> </li>

</ul> <p> <b>UTF-8</b>  </p> <p>UTF-8 encodes and transmits Unicode characters

as a string of 8-bit bytes. All the ASCII characters 0 to 127 are encoded

without change; the most significant bit being set to zero is a signal that

they have not been changed. Unicode characters U0080 to U07FF are encoded

in two bytes, the remaining Unicode characters — except for the surrogates —

are encoded in three bytes. The Unicode surrogate characters are supported

by the Character Conversion API, but are not currently supported by all Symbian

platform components. </p> <p>A variant of UTF-8 used internally by Java differs

from standard UTF-8 in two ways. First, the specific case of the NULL character

(0x0000) is encoded in the two-byte format, and second, only the one-, two-

and three-byte formats are used, not the four-byte format which is normally

used for Unicode surrogate-pairs. An argument to <codeph>ConvertFromUnicodeToUtf8</codeph> controls

whether the UTF-8 generated by this is the Java variant. Support for this

was removed in v6.0. </p> </section>

<section id="GUID-5E593C5A-882B-5B11-AD6E-CFD10EA6700B"><title>Charconv converter</title> <p>Each

converter implements a conversion between a single foreign character encoding

and UTF-16, and is identified by a Unique Identifier (UID). The Symbian platform

provides the following two types of converter: </p> <ul>

<li id="GUID-8A1AFA7F-E330-5309-9ED7-26A4A46411CB"><p>built into the Framework

component and used by most languages </p> </li>

<li id="GUID-E7A3FD30-37E0-5B01-B2BE-7DE313045D9F"><p>implemented as Ecom

plug-ins in the Plug-ins component and used by certain languages. </p> </li>

</ul> </section>

</conbody></concept>