Converting text between different character-encoding systems
EPOC machines internally store text encoded in Unicode - the character set defined by The Unicode Consortium. However, much of the text sent between the EPOC machine and the outside world (e.g. in emails) will be encoded in character sets other than Unicode. The CHARCONV component provides a DLL for converting text (both ways) between Unicode and other standard characters sets. It also supports converting (both ways) between UCS-2 and UTF-7, as well as between UTF-16 and UTF-8. There is no direct way to convert between UTF-7 and UTF-8 though - in fact CHARCONV as a whole does not provide any direct conversion facilities other than to or from Unicode (UCS-2 or UTF-16).
Converting text between Unicode and other standard character sets
The class CCnvCharacterSetConverter is provided for converting text between Unicode and other standard character sets. The public parts of the class are shown below:

class CCnvCharacterSetConverter : public CBase
	{
public:
	enum TAvailability
		{
		EAvailable,
		ENotAvailable
		};
	enum TError
		{
		EErrorIllFormedInput=-1
		};
	enum TEndianness
		{
		ELittleEndian,
		EBigEndian
		};
	enum {KStateDefault=0};
	struct SCharacterSet
		{
		inline TUint Identifier() const;
		inline TBool NameIsFileName() const;
		inline TPtrC Name() const;
	private:
		// ...
		};
public:
	IMPORT_C static CCnvCharacterSetConverter* NewL();
	IMPORT_C static CCnvCharacterSetConverter* NewLC();
	IMPORT_C virtual ~CCnvCharacterSetConverter();
	IMPORT_C static CArrayFix<SCharacterSet>* CreateArrayOfCharacterSetsAvailableL(RFs&);
	IMPORT_C static CArrayFix<SCharacterSet>* CreateArrayOfCharacterSetsAvailableLC(RFs&);
	IMPORT_C TUint ConvertStandardNameOfCharacterSetToIdentifierL(const TDesC8&, RFs&);
	IMPORT_C HBufC8* ConvertCharacterSetIdentifierToStandardNameL(TUint, RFs&);
	IMPORT_C TUint ConvertMibEnumOfCharacterSetToIdentifierL(TInt, RFs&);
	IMPORT_C TInt ConvertCharacterSetIdentifierToMibEnumL(TUint, RFs&);
	IMPORT_C void PrepareToConvertToOrFromL(TUint, const CArrayFix<SCharacterSet>&, RFs&);
	IMPORT_C TAvailability PrepareToConvertToOrFromL(TUint, RFs&);
	IMPORT_C void SetDefaultEndiannessOfForeignCharacters(TEndianness);
	IMPORT_C void SetReplacementForUnconvertibleUnicodeCharactersL(const TDesC8&);
	IMPORT_C TInt ConvertFromUnicode(TDes8&, const TDesC16&) const;
	IMPORT_C TInt ConvertFromUnicode(TDes8&, const TDesC16&, TInt&) const;
	IMPORT_C TInt ConvertFromUnicode(TDes8&, const TDesC16&, TInt&, TInt&) const;
	IMPORT_C TInt ConvertToUnicode(TDes16&, const TDesC8&, TInt&) const;
	IMPORT_C TInt ConvertToUnicode(TDes16&, const TDesC8&, TInt&, TInt&) const;
	IMPORT_C TInt ConvertToUnicode(TDes16&, const TDesC8&, TInt&, TInt&, TInt&) const;
private:
	// ...
	};
Overview
There are two basic stages involved in using a CCnvCharacterSetConverter object to change the encoding of a piece of text from one character set to another. It is first necessary to select the foreign character set of interest. Then having done that, the actual conversion of text to or from that character set can be performed. Selecting the foreign character set is done by calling one of the overloads of PrepareToConvertToOrFromL. Text conversion itself is done by calling one of the overloads of ConvertFromUnicode or ConvertToUnicode.
Creating and destroying CCnvCharacterSetConverter objects
Objects of the CCnvCharacterSetConverter class are created by calling either of the static member functions NewL or NewLC, the latter leaving the object on the cleanup stack. These objects are created on the heap and are thus destroyed using the delete operator as normal. Creating one of these objects only involves some allocation on the heap and initialization of member data - it does not involve anything as onerous as scanning for files, or loading data from a file. Such “heavyweight” activities are done at a later stage (see “Selecting the foreign character set to convert to/from” below).
The member functions for doing the conversion
The most important member functions of the class are ConvertFromUnicode and ConvertToUnicode. These all convert the text passed in the second parameter and output the resulting text in the first parameter. Sixteen-bit descriptors are used to hold any text encoded in Unicode (UCS-2 or UTF-16), and eight-bit descriptors are used to hold any text encoded in any other character set. Foreign character sets (i.e. character sets other than Unicode) may use a single byte per character (e.g. Code Page 1252) or more than one byte per character (e.g. China’s GB 2312-80) or even a variable number of bytes per character (e.g. Japan’s Shift-JIS) - hence the need to use eight-bit descriptors.
The third parameter to the second two overloads of ConvertFromUnicode (which is purely an output parameter) holds the indices of each Unicode character in the source text that could not be converted.  This is because the foreign character set has no equivalents for these Unicode characters. It can record, at a maximum, 25 unconvertible characters. So if the input source text has more than 25 unconvertible characters, only the first 25 of these characters will be recorded.   Similarly, the fourth parameters to the second two overloads of ConvertToUnicode (which are also purely output parameters) tell the caller the number of characters in the input descriptor that could not be converted into Unicode (for the analogous reason). In neither ConvertFromUnicode or ConvertToUnicode is conversion aborted because of unconvertible characters. (What is written to the output descriptor for such unconvertible characters is discussed in a separate paragraph below.) The purpose of these parameters in the case of both ConvertFromUnicode and ConvertToUnicode is to allow the caller to gauge a rough idea of the proportion of characters that are failing to be converted. (Recall that a character in the input descriptor to ConvertToUnicode does not necessarily occupy only a single byte of that descriptor. However, since Unicode is intended to cover all possible characters and thus be the superset of all character sets, this will rarely, if ever, report anything other than zero characters, although it will report the existence of unconvertible characters if the input descriptor contains illegal “characters”, i.e. values not actually in the foreign character set.)
The last parameter to the last overloads of both ConvertFromUnicode and ConvertToUnicode (these are again purely output parameters) indicates the index of the first character (or in the case of ConvertToUnicode, the index of the first byte of the first character) in the input descriptor that could not be converted. If all the characters in the input descriptor could be converted, this parameter is set to a negative value.
The TInt value returned by ConvertFromUnicode and ConvertToUnicode is either one of the negative error values defined in CCnvCharacterSetConverter’s enum TError or the number of elements left at the end of the input descriptor which were not converted, either (i) because there was no room left in the output descriptor, or (ii) because the input descriptor ended with an incomplete sequence (e.g. half-way through a multi-byte character).
•	The ability to do partial conversions means that the CHARCONV client does not have to try to guess how big to make the output descriptor for a given input descriptor, they can simply do the conversion in a loop using a smallish output descriptor (an output descriptor whose MaxLength() is less than about 20 elements would not be recommended) - the code in the “Sample code” section below gives a demonstration of how to do this. It is worth noting that even if the output descriptor was not big enough to hold the entire converted input text, it will not necessarily be full in the sense of have its Length() equal its MaxLength(). The only sure way to find out whether the whole input descriptor has been converted is to test whether the return value (if not negative) is zero or not.
•	The ability to cope with an input descriptor being truncated is useful if the client cannot guarantee whether the input descriptor will be complete, e.g. if they themselves are receiving it in chunks from an external source. All that the client needs to do to handle this situation is to add the specified number of descriptor elements to the start of the input descriptor passed to the next call. However, there is a binary compatibility issue here. For those clients whose input descriptor is known to be complete, if they perform the sort of loop found in the first two examples in the “Sample code” section, they would get into an infinite loop if the input descriptor was corrupt such that it ended with an incomplete sequence, as the conversion function would always return a value greater than zero. To avoid this problem arising, the behavior of the conversion functions is actually such that if none of the input descriptor can be consumed because it consists exclusively of an incomplete sequence, the error EErrorIllFormedInput (defined in CCnvCharacterSetConverter’s enum TError) will be returned. In order for this error not to be returned when the presence of an incomplete sequence at the end of the input descriptor does not mean it is corrupt, the input descriptor must be long enough such that it cannot consist exclusively of an incomplete sequence - an input descriptor of at least 20 elements should be plenty long enough. (The third example in the “Sample code” section below gives an illustration of how to use CHARCONV to convert fragmented input.)
The third parameter to ConvertToUnicode is used to save state information across multiple calls to ConvertToUnicode where each call starts off where the previous call reached in the input descriptor. At the start of each such series of calls to ConvertToUnicode this parameter must be set to CCnvCharacterSetConverter::KStateDefault and thereafter not tampered with, but simply passed in to each ConvertToUnicode call.
Selecting the foreign character set to convert to/from
When the CCnvCharacterSetConverter object has been created, before either ConvertFromUnicode or ConvertToUnicode can be called, it is necessary to state which foreign character set is to be converted to/from. This is done by calling either of the two overloads of PrepareToConvertToOrFromL. Each takes the identifier (a Symbian-defined UID) of the required character set as it’s first parameter, and a file-server session object as it’s last parameter. The first overload requires the array returned by either of CCnvCharacterSetConverter’s static member functions CreateArrayOfCharacterSetsAvailableL or CreateArrayOfCharacterSetsAvailableLC to be passed in as its second parameter. This array contains one CCnvCharacterSetConverter::SCharacterSet item for each foreign character set that is available (at run-time) for text conversion. The Identifier member function of SCharacterSet returns the Symbian-UID identifier of the character set represented by that object. (See below for a discussion of SCharacterSet’s other member functions.) Creating this array is not required for calling the second overload.
If the first overload of PrepareToConvertToOrFromL is used, the character-set identifier passed into the first parameter must correspond to an item in the array passed to the second parameter (i.e. that item’s Identifier member function returns the same number), or a panic will occur. The second overload, however, is less demanding - the character set identified in the first parameter does not have to be available at run-time. Depending on whether the character set is available or not, it returns EAvailable or ENotAvailable (from CCnvCharacterSetConverter’s enum TAvailability). Calls to ConvertFromUnicode or ConvertToUnicode convert to or from the character set whose identifier was passed into the last PrepareToConvertToOrFromL call which did not return ENotAvailable. If all calls previously returned ENotAvailable, a panic will occur.
The two overloads of PrepareToConvertToOrFromL are designed for different usage. In a situation where it may be necessary to present the user with a list of all the possible character sets for them to choose which to convert to/from, then the first overload would need to be used. However, if the CHARCONV client wishes to convert the content of, say, an email message, or an HTML document, i.e. something that describes within itself what character set its content is in, then the second overload would be more appropriate.
Efficiency considerations
The first overload of PrepareToConvertToOrFromL should be used in preference to the second overload if a number of different character sets are to be converted to/from in a short space of time. The reason for this is that the second overload, unlike the first, internally has to do a file scan which is similar to the one performed by CreateArrayOfCharacterSetsAvailableL and CreateArrayOfCharacterSetsAvailableLC. Thus it is more efficient to do the file-scanning once to start with by calling one of these array-creating functions, and then pass that array in to the first overload of PrepareToConvertToOrFromL each time the target/source character set is to be changed (first checking to see if each required character set is in the array, to avoid the panic that occurs if it is not). However, if the different calls to PrepareToConvertToOrFromL are spread out over a considerable time, it may be best to use the second overload for this as it will perform the necessary re-scanning of files each time and thus pick up any new files that have been installed in the mean time.
SCharacterSet and the names of foreign character sets
SCharacterSet’s member function Name returns the full path and file name of the conversion-data file for that character set, although for those character sets for which CHARCONV hard codes conversion data (rather than getting it from a file) Name returns an arbitrary name for that character set. Note that the name returned cannot be treated as an Internet-standard name. Converting between the Internet-standard names of character sets and the Symbian-UID identifiers that PrepareToConvertToOrFromL expects is provided by the member functions ConvertStandardNameOfCharacterSetToIdentifierL and ConvertCharacterSetIdentifierToStandardNameL (see “Other member functions” below). Also, the name returned is locale-independent and therefore should be hidden from the user. NameIsFileName returns whether Name returns a file name or just an arbitrary name. (This may prove useful, for example, to find out whether F32’s file-name parsing class TParse can be used on the return value of Name.)
Other member functions
For character sets using more than one byte per character, the “endian”-ness of those characters may be defined or left open. Although none of the foreign character sets currently supported by CHARCONV are of undefined endianness, the member function SetDefaultEndiannessOfForeignCharacters which takes either ELittleEndian or EBigEndian (from CCnvCharacterSetConverter’s enum TEndianness) allows the CHARCONV client to set the endianness that ConvertFromUnicode and ConvertToUnicode use when converting to or from foreign character sets of undefined endianness.
When converting to Unicode, each character in the input descriptor for which there is no equivalent in Unicode is converted to the Unicode character specifically designated for that purpose - 0xFFFD. (As already noted, this should rarely, if ever, be the case, unless the input is corrupt.) When converting the other way, each foreign character set provides a default character to be used for each Unicode character lacking an equivalent in that character set. This default may be overridden by calling the member function SetReplacementForUnconvertibleUnicodeCharactersL. It should be noted that PrepareToConvertToOrFromL undoes the effect of any previous SetReplacementForUnconvertibleUnicodeCharactersL call, and thus for the latter to have any effect, it should be called between the PrepareToConvertToOrFromL call and the subsequent ConvertFromUnicode call(s). The parameter to SetReplacementForUnconvertibleUnicodeCharactersL should be passed a single, possibly multi-byte, character, whose byte order (if multi-byte) should be little-endian if the endianness of the character set is undefined, otherwise the same endianness as is defined by the character set.
ConvertStandardNameOfCharacterSetToIdentifierL is provided for the CHARCONV client to be able to find out the Symbian UID of a character set for a given Internet-standard name of that character set. It returns zero if the name of the character set was not known. This function can be called at any time in the CCnvCharacterSetConverter object’s lifetime.
ConvertCharacterSetIdentifierToStandardNameL is provided for the CHARCONV client to be able to find out an Internet-standard name of a character set (which is a “preferred MIME name” where possible) for the Symbian UID of that character set. It returns NULL if the UID of the character set was not known. This function can be called at any time in the CCnvCharacterSetConverter object’s lifetime.
ConvertMibEnumOfCharacterSetToIdentifierL is provided for the CHARCONV client to be able to find out the Symbian UID of a character set for a given MIB-enum of that character set. It returns zero if the name of the character set was not known. This function can be called at any time in the CCnvCharacterSetConverter object’s lifetime.
ConvertCharacterSetIdentifierToMibEnumL is provided for the CHARCONV client to be able to find out a MIB-enum of a character set for the Symbian UID of that character set. It returns zero if the UID of the character set was not known. This function can be called at any time in the CCnvCharacterSetConverter object’s lifetime.
The four functions described above all behave as follows. The first time any if them is called with a first parameter that is not in CCnvCharacterSetConverter’s internal hard-coded list, a file scan is done to try to find out the required information from a file. The results of this are stored in the CCnvCharacterSetConverter object, so that no subsequent call to any of these four functions will repeat the file scanning.
Sample code
The sample code below illustrates typical usage of CCnvCharacterSetConverter.

An example of how to convert, in small chunks, Unicode text to another character set:

LOCAL_C void ConvertUnicodeTextL(CCnvCharacterSetConverter& aCharacterSetConverter,
								RFs& aFileServerSession,
								TUint aForeignCharacterSet,
								const TDesC16& aUnicodeText)
	{
	if (aCharacterSetConverter.PrepareToConvertToOrFromL(aForeignCharacterSet, aFileServerSession)
														!=CCnvCharacterSetConverter::EAvailable)
		User::Leave(KErrNotSupported);

	TBuf8<20> outputBuffer;
	TPtrC16 remainderOfUnicodeText(aUnicodeText);
	FOREVER
		{
		const TInt returnValue=aCharacterSetConverter.ConvertFromUnicode(outputBuffer,
																		remainderOfUnicodeText);
		if (returnValue==CCnvCharacterSetConverter::EErrorIllFormedInput)
			User::Leave(KErrCorrupt);
		else if (returnValue<0) // future-proof against "TError" expanding
			User::Leave(KErrGeneral);

		// ? - do something here with outputBuffer

		if (returnValue==0)
			break; // all of aUnicodeText has been converted and handled
		remainderOfUnicodeText.Set(remainderOfUnicodeText.Right(returnValue));
		}
	}

An example of how to convert, in small chunks, text in another character set to Unicode:

LOCAL_C void ConvertForeignTextL(CCnvCharacterSetConverter& aCharacterSetConverter,
								RFs& aFileServerSession,
								TUint aForeignCharacterSet,
								const TDesC8& aForeignText)
	{
	if (aCharacterSetConverter.PrepareToConvertToOrFromL(aForeignCharacterSet, aFileServerSession)
														!=CCnvCharacterSetConverter::EAvailable)
		User::Leave(KErrNotSupported);

	TBuf16<20> outputBuffer;
	TPtrC8 remainderOfForeignText(aForeignText);
	// "state" must be initialized to CCnvCharacterSetConverter::KStateDefault - it must not be
	// tampered with thereafter but simply passed into each call of "ConvertToUnicode"
	TInt state=CCnvCharacterSetConverter::KStateDefault;
	FOREVER
		{
		const TInt returnValue=aCharacterSetConverter.ConvertToUnicode(outputBuffer,
																	remainderOfForeignText,
																	state);
		if (returnValue==CCnvCharacterSetConverter::EErrorIllFormedInput)
			User::Leave(KErrCorrupt);
		else if (returnValue<0) // future-proof against "TError" expanding
			User::Leave(KErrGeneral);

		// ? - do something here with outputBuffer

		if (returnValue==0)
			break; // all of aForeignText has been converted and handled
		remainderOfForeignText.Set(remainderOfForeignText.Right(returnValue));
		}
	}

An example of how to convert, in small chunks, fragmented text in another character set to Unicode:

LOCAL_C void ConvertForeignTextL(CCnvCharacterSetConverter& aCharacterSetConverter,
								RFs& aFileServerSession,
								TUint aForeignCharacterSet)
	{
	if (aCharacterSetConverter.PrepareToConvertToOrFromL(aForeignCharacterSet, aFileServerSession)
														!=CCnvCharacterSetConverter::EAvailable)
		User::Leave(KErrNotSupported);

	const TInt KMaximumLengthOfBufferForForeignText=200;
	TUint8 bufferForForeignText[KMaximumLengthOfBufferForForeignText];
	TInt lengthOfBufferForForeignText=0;

	FOREVER
		{
		TPtr8 nextChunkOfForeignText(bufferForForeignText+lengthOfBufferForForeignText,
							KMaximumLengthOfBufferForForeignText-lengthOfBufferForForeignText);

		// ? load nextChunkOfForeignText here

		const TInt lengthOfNextChunkOfForeignText=nextChunkOfForeignText.Length();
		lengthOfBufferForForeignText+=lengthOfNextChunkOfForeignText;

		const TBool isLastChunkOfForeignText= // ? find out from the source of the foreign text
		// whether this is the last chunk and set this variable accordingly, e.g. the source may
		// define that the last chunk is of length zero, in which case the expression
		// "(lengthOfNextChunkOfForeignText==0)" would be assigned to this variable; note that
		// even if the length of this chunk is zero, we can't just exit this function here as
		// bufferForForeignText may not be empty (i.e. lengthOfBufferForForeignText>0)

		TBuf16<20> outputBuffer;
		TPtrC8 remainderOfForeignText(bufferForForeignText, lengthOfBufferForForeignText);
		// "state" must be initialized to CCnvCharacterSetConverter::KStateDefault - it must not
		// be tampered with thereafter but simply passed into each call of "ConvertToUnicode"
		TInt state=CCnvCharacterSetConverter::KStateDefault;
		FOREVER
			{
			const TInt lengthOfRemainderOfForeignText=remainderOfForeignText.Length();
			if (isLastChunkOfForeignText)
				{
				if (lengthOfRemainderOfForeignText==0)
					return; // the single point of exit of this function
				}
			else
				{
				// As this isn't the last chunk, we don't want ConvertToUnicode to return
				// CCnvCharacterSetConverter::EErrorIllFormedInput if the input descriptor ends
				// with an incomplete sequence - but it will only do this if *none* of the input
				// descriptor can be consumed. Therefore if the input descriptor is long enough
				// (20 elements or longer is plenty adequate) there is no danger of this error
				// being returned for this reason. If it's shorter than that, we'll simply put it
				// at the start of the buffer so that it gets converted with the next chunk.
				if (lengthOfRemainderOfForeignText<20)
					{
					// put any remaining foreign text at the start of bufferForForeignText
					lengthOfBufferForForeignText=lengthOfRemainderOfForeignText;
					Mem::Copy(bufferForForeignText, remainderOfForeignText.Ptr(),
													lengthOfBufferForForeignText);
					break;
					}
				}

			const TInt returnValue=aCharacterSetConverter.ConvertToUnicode(outputBuffer,
																		remainderOfForeignText,
																		state);
			if (returnValue==CCnvCharacterSetConverter::EErrorIllFormedInput)
				User::Leave(KErrCorrupt);
			else if (returnValue<0) // future-proof against "TError" expanding
				User::Leave(KErrGeneral);

			// ? - do something here with outputBuffer

			remainderOfForeignText.Set(remainderOfForeignText.Right(returnValue));
			}
		}
	}
Converting text between UCS-2/UTF-16 and Unicode’s transformation formats
The class CnvUtfConverter is provided for converting text between UCS-2 and UTF-7, as well as between UTF-16 and UTF-8. (For the definition of these formats, see The Unicode Standard published by The Unicode Consortium.) The public parts of the class are shown below:

class CnvUtfConverter
	{
public:
	enum TError
		{
		EErrorIllFormedInput=-1
		};
	enum {KStateDefault=0};
public:
	IMPORT_C static TInt ConvertFromUnicodeToUtf7(TDes8&, const TDesC16&, TBool);
	IMPORT_C static TInt ConvertFromUnicodeToUtf8(TDes8&, const TDesC16&);
	IMPORT_C static TInt ConvertToUnicodeFromUtf7(TDes16&, const TDesC8&, TInt&);
	IMPORT_C static TInt ConvertToUnicodeFromUtf8(TDes16&, const TDesC8&);
private:
	// ...
	};

In fact the functionality of this class is also available via the CCnvCharacterSetConverter class by passing it the UID for UTF-7 or UTF-8. However, CnvUtfConverter is provided for ease of use as it requires no instantiation (all the member functions are static). The four functions all convert the text passed in the second parameter and output the resulting text in the first parameter. Sixteen-bit descriptors are used to hold text encoded in UCS-2 or UTF-16, and eight-bit descriptors are used to hold text encoded either of the transformation formats.
UTF-7 divides the Unicode character set into 3 subsets: set D which is the set of characters that are encoded directly (i.e. unaltered), set B which is the set of characters that are encoded in modified “base 64”, and set O which is the set of characters that can either be encoded directly, or in modified “base 64”. The TBool parameter to ConvertFromUnicodeToUtf7 controls the way that the characters in set O are encoded: passing in ETrue causes them to be encoded in “base 64” (which is the “safer” option), and EFalse causes them to encoded directly. The contents of set O are listed in their entirety below.

!"#$%&*;<=>@[]^_`{|}

The variant of UTF-8 used internally by Java differs from standard UTF-8 in two ways. First, the specific case of the NULL character (0x0000) is encoded in the two-byte format, and second, only the one-, two- and three-byte formats are used (i.e. not the four-byte format which is normally used for Unicode surrogate-pairs). Generating this variant of UTF-8 is not supported via CnvUtfConverter; however, it is possible by using CCnvCharacterSetConverter with a UID of KCharacterSetIdentifierJavaConformantUtf8.
The TInt value returned by all four member functions is either one of the negative error values defined in CnvUtfConverter’s enum TError or the number of elements left at the end of the input descriptor which were not converted, either (i) because there was no room left in the output descriptor, or (ii) because the input descriptor ended with an incomplete sequence (e.g. half-way through a multi-byte character).
•	The ability to do partial conversions means that the CHARCONV client does not have to try to guess how big to make the output descriptor for a given input descriptor, they can simply do the conversion in a loop using a smallish output descriptor (an output descriptor whose MaxLength() is less than about 20 elements would not be recommended) - the code in the “Sample code” section below gives a demonstration of how to do this. It is worth noting that even if the output descriptor was not big enough to hold the entire converted input text, it will not necessarily be full in the sense of have its Length() equal its MaxLength(). The only sure way to find out whether the whole input descriptor has been converted is to test whether the return value (if not negative) is zero or not.
•	The ability to cope with an input descriptor being truncated is useful if the client cannot guarantee whether the input descriptor will be complete, e.g. if they themselves are receiving it in chunks from an external source. All that the client needs to do to handle this situation is to add the specified number of descriptor elements to the start of the input descriptor passed to the next call. However, there is a binary compatibility issue here. For those clients whose input descriptor is known to be complete, if they perform the sort of loop found in the first two examples in the “Sample code” section, they would get into an infinite loop if the input descriptor was corrupt such that it ended with an incomplete sequence, as the conversion function would always return a value greater than zero. To avoid this problem arising, the behavior of the conversion functions is actually such that if none of the input descriptor can be consumed because it consists exclusively of an incomplete sequence, the error EErrorIllFormedInput (defined in CCnvCharacterSetConverter’s enum TError) will be returned. In order for this error not to be returned when the presence of an incomplete sequence at the end of the input descriptor does not mean it is corrupt, the input descriptor must be long enough such that it cannot consist exclusively of an incomplete sequence - an input descriptor of at least 20 elements should be plenty long enough. (The third example in the “Sample code” section below gives an illustration of how to use CHARCONV to convert fragmented input.)
The third parameter to ConvertToUnicodeFromUtf7 is used to save state information across multiple calls to ConvertToUnicodeFromUtf7 where each call starts off where the previous call reached in the input descriptor. At the start of each such series of calls to ConvertToUnicodeFromUtf7, this parameter must be set to CnvUtfConverter::KStateDefault and thereafter not tampered with, but simply passed in to each ConvertToUnicode call.
Sample Code
The sample code below illustrates typical usage of CnvUtfConverter to convert to and from UTF-7. (Converting to and from UTF-8 is almost identical, the only differences being that ConvertFromUnicodeToUtf8 and ConvertToUnicodeFromUtf8 each only take two parameters.)

An example of how to convert, in small chunks, Unicode (UCS-2) to UTF-7:

LOCAL_C void EncodeL(const TDesC16& aUnicodeText)
	{
	TBuf8<20> outputBuffer;
	TPtrC16 remainderOfUnicodeText(aUnicodeText);
	FOREVER
		{
		const TInt returnValue=CnvUtfConverter::ConvertFromUnicodeToUtf7(outputBuffer,
																		remainderOfUnicodeText,
ETrue);
		if (returnValue==CnvUtfConverter::EErrorIllFormedInput)
			User::Leave(KErrCorrupt);
		else if (returnValue<0) // future-proof against "TError" expanding
			User::Leave(KErrGeneral);

		// ? - do something here with outputBuffer

		if (returnValue==0)
			break; // all of aUnicodeText has been converted and handled
		remainderOfUnicodeText.Set(remainderOfUnicodeText.Right(returnValue));
		}
	}

An example of how to convert, in small chunks, UTF-7 to Unicode (UCS-2):

LOCAL_C void DecodeL(const TDesC8& aUtf7)
	{
	TBuf16<20> outputBuffer;
	TPtrC8 remainderOfUtf7(aUtf7);
	// "state" must be initialized to CnvUtfConverter::KStateDefault - it must not be tampered
// with thereafter but simply passed into each call of "ConvertToUnicodeFromUtf7"
	TInt state=CnvUtfConverter::KStateDefault;
	FOREVER
		{
		const TInt returnValue=CnvUtfConverter::ConvertToUnicodeFromUtf7(outputBuffer,
remainderOfUtf7,
state);
		if (returnValue==CnvUtfConverter::EErrorIllFormedInput)
			User::Leave(KErrCorrupt);
		else if (returnValue<0) // future-proof against "TError" expanding
			User::Leave(KErrGeneral);

		// ? - do something here with outputBuffer

		if (returnValue==0)
			break; // all of aUtf7 has been converted and handled
		remainderOfUtf7.Set(remainderOfUtf7.Right(returnValue));
		}
	}

An example of how to convert, in small chunks, fragmented UTF-7 to Unicode (UCS-2):

LOCAL_C void DecodeL()
	{
	const TInt KMaximumLengthOfBufferForUtf7=200;
	TUint8 bufferForUtf7[KMaximumLengthOfBufferForUtf7];
	TInt lengthOfBufferForUtf7=0;

	FOREVER
		{
		TPtr8 nextChunkOfUtf7(bufferForUtf7+lengthOfBufferForUtf7,
							KMaximumLengthOfBufferForUtf7-lengthOfBufferForUtf7);

		// ? load nextChunkOfUtf7 here

		const TInt lengthOfNextChunkOfUtf7=nextChunkOfUtf7.Length();
		lengthOfBufferForUtf7+=lengthOfNextChunkOfUtf7;

		const TBool isLastChunkOfUtf7= // ? find out from the source of the UTF-7 whether this is
		// the last chunk and set this variable accordingly, e.g. the source may define that the
		// last chunk is of length zero, in which case the expression
		// "(lengthOfNextChunkOfUtf7==0)" would be assigned to this variable; note that even if
		// the length of this chunk is zero, we can't just exit this function here as
		// bufferForUtf7 may not be empty (i.e. lengthOfBufferForUtf7>0)

		TBuf16<20> outputBuffer;
		TPtrC8 remainderOfUtf7(bufferForUtf7, lengthOfBufferForUtf7);
		// "state" must be initialized to CnvUtfConverter::KStateDefault - it must not be tampered
		// with thereafter but simply passed into each call of "ConvertToUnicodeFromUtf7"
		TInt state=CnvUtfConverter::KStateDefault;
		FOREVER
			{
			const TInt lengthOfRemainderOfUtf7=remainderOfUtf7.Length();
			if (isLastChunkOfUtf7)
				{
				if (lengthOfRemainderOfUtf7==0)
					return; // the single point of exit of this function
				}
			else
				{
				// As this isn't the last chunk, we don't want ConvertToUnicodeFromUtf7 to return
				// CnvUtfConverter::EErrorIllFormedInput if the input descriptor ends with an
				// incomplete sequence - but it will only do this if *none* of the input
				// descriptor can be consumed. Therefore if the input descriptor is long enough
				// (20 elements or longer is plenty adequate) there is no danger of this error
				// being returned for this reason. If it's shorter than that, we'll simply put it
				// at the start of the buffer so that it gets converted with the next chunk.
				if (lengthOfRemainderOfUtf7<20)
					{
					// put any remaining UTF-7 at the start of bufferForUtf7
					lengthOfBufferForUtf7=lengthOfRemainderOfUtf7;
					Mem::Copy(bufferForUtf7, remainderOfUtf7.Ptr(), lengthOfBufferForUtf7);
					break;
					}
				}

			const TInt returnValue=CnvUtfConverter::ConvertToUnicodeFromUtf7(outputBuffer,
																			remainderOfUtf7,
																			state);
			if (returnValue==CnvUtfConverter::EErrorIllFormedInput)
				User::Leave(KErrCorrupt);
			else if (returnValue<0) // future-proof against "TError" expanding
				User::Leave(KErrGeneral);

			// ? - do something here with outputBuffer

			remainderOfUtf7.Set(remainderOfUtf7.Right(returnValue));
			}
		}
	}

A description of the UTF-8 conversion routines:

Example usage of CnvUtfConverter::ConvertFromUnicodeToUtf8(TDes8&, const TDesC16&):

The conversion from Unicode (UTF-16) to UTF-8.

1.	ConvertFromUnicodeToUtf8(TDes8&, const TDesC16&) can accept valid 
surrogate pairs (code points > U+FFFF), each surrogate pair encountered will output a valid 4 byte UTF-8 sequence. 

For example:

	The UCS2 input of: 0xd800 0xdc00 (a surrogate pair)

	Will output UTF-8: 0xf0 0x90 0x80 0x80 (a valid 4 byte UTF-8 string)

2.	The converter when used via CCnvCharacterSetConverter with a UID of KCharacterSetIdentifierJavaConformantUtf8 will be java compliant.

For example:

	The UCS2 input of: 0x0000 0xd9e0 0xdda3 

	Will output UTF-8: 0xc0 0x80 0xe8 0x87 0xaa

Example usage of CnvUtfConverter::ConvertToUnicodeFromUtf8(TDes16&, const TDesC8&) which is now standards compliant:

For reference table 3-7 from “The Unicode Standard 5.0”

Code Points	1st Byte	2nd Byte	3rd Byte	4th Byte	Description	
U+0000..U+007F	00..7D				1 byte ascii	
U+0800..U+07FF	C2..DF	80..BF			2 bytes, error if 1st > 0xC2	
U+1000..U+CFFF	E1..EC	80..BF	80..BF		Normal	
U+D000..U+D7FF	ED	80..9F	80..BF		3 bytes, 1st ==0xED, error if 2nd > 0x9F	
U+E000..U+FFFF	EE..EF	80..BF	80..BF		Normal	
U+10000..U+3FFFF	F0	90..BF	80..BF	80..BF	4 bytes, 1st == 0xF0, error if 2nd < 0x90	
U+10000..U+3FFFF	F1..F3	80..BF	80..BF	80..BF	Normal	
U+100000..U+10FFFF	F4	80..8F	80..BF	80..BF	4 bytes, 1st == 0xF4, error if 2nd > 0x8F	

The conversion from UTF-8 to Unicode (UTF-16).

1.	If the conversion routine encounters an invalid UTF-8 sequence of 2-4 bytes in length (i.e. the flag at the first byte signals that this is a 2-4 byte sequence), the conversion routine will start at the next byte in the sequence.

For example:

	The UTF-8 input of: 0xf4, 0x20, 0x01, 0x02, 0x10

Will output in UTF-16: 0xFFFD, 0x0020, 0x0001, 0x0002, 0x0010, because 0xf4, 0x20, 0x01, 0x02 is not a valid 4 byte UTF-8 sequence and the next byte is read.

2.	This conversion routine is standards compliant as it will not accept a valid a bad input UTF-8 sequence e.g. 0xc1 0x82.

For example:

The UTF-8 input of: 0xc1 0x82

Will output in UTF-16: 0xFFFD 0xFFFD, as 0xc1 isn’t a valid two byte sequence (must start at 0xC2, see table above) and neither is 0x82 (again must start at 0xc2).

3.	The UTF-8 converter follows all rules for UTF-8 conversion.  Particular attention should be paid to D92 in section 3.9 from “The Unicode Standard 5.0” that explicitly states:

•	Any UTF-8 byte sequence that does not match the patterns listed in Table 3-7 is ill-formed.
•	Before the Unicode Standard, Version 3.1, the problematic “non-shortest form”
byte sequences in UTF-8 were those where BMP characters could be represented
in more than one way. These sequences are ill-formed, because they are not allowed by Table 3-7.

4.	CnvUtfConverter::ConvertToUnicodeFromUtf8(TDes16&, const TDesC8&) can create surrogate pairs (code points > U+FFFF) when converting a valid 4 byte UTF-8 sequence.

For example:

	The UTF-8 input of: 0xf0 0x90 0x80 0x80 (a valid 4 byte)

	Will output in UTF-16: 0xd800 0xdc00 (a surrogate pair)

5.	The converter when used via CCnvCharacterSetConverter with a UID of KCharacterSetIdentifierJavaConformantUtf8 will be java compliant.

For example:

	The UTF-8 input of: 0xc0 0x80 0xe8 0x87 0xaa

	Will output in UTF-16: 0x0000 0xd9e0 0xdda3 (as otherwise without KCharacterSetIdentifierJavaConformantUtf8 output is 0xFFFD 0xFFFD 0xd9e0 0xdda3)