EPOC Charconv - Autodetection 

Author: Arunsakh Sachamuneewongse
Contributors: David Batchelor. 
Date: 19th October 2000 

Introduction

Charconv
Charconv seemed the ideal place to implement the new Autodetect functionality, as autodetect can use Charconv’s information regarding all the different character sets. Since autodetect is implemented in Charconv, the autodetect framework is able to detect Charconv’s built-in character sets, which are ASCII, SMS7Bit, CP1252, ISO 88591, UTF7 & UTF8. Besides these character sets it also provides support framework for Charconv’s plug-ins character sets such as GBK, SHIFT-JIS, ISO2022JP, ISO 88592 - 99, although only the Japanese and the Chinese character sets have been implemented in autodetect for 6.1. 

The following static function has been added to CCnvCharacterSetConverter 

IMPORT_C static void CCnvCharacterSetConverter::AutoDetectCharacterSetL(
	TInt& aConfidenceLevel,
	TUint& aCharacterSetIdentifier,
	const CArrayFix<SCharacterSet>& aArrayOfCharacterSetsAvailable,
	const TDesC8& aSample)

The first two parameters are both output-only parameters: aConfidenceLevel is set by the function to a value between 0 and 100 (inclusive), where 0 means "I have no idea what character set aSample is in" (in which case aCharacterSetIdentifier is undefined), and 100 means "I have total confidence that aCharacterSetIdentifier is the character-set of aSample". The aArrayOfCharacterSetsAvailable parameter is included for efficiency - CCnvCharacterSetConverter::AutoDetectCharacterSetL could itself find the list of character-sets available but this is a very time-consuming task and the client may already have this information to hand.

The Charconv framework provides support for a few character sets built into it. These character sets are ASCII, SMS7Bit, CP1252, ISO 88591, UTF7 & UTF8. The framework also provides support for many other character sets. This support is provided by Plug-in DLLs. At the CHARCONV plug-in DLL interface, the first reserved function (at ordinal 4) will be replaced with a new exported function as follows:

IMPORT_C TBool IsInThisCharacterSetL(
	TBool& aSetToTrue,
	TInt& aConfidenceLevel,
	const TDesC8& aSample)

The aSetToTrue parameter is just to indicate to CCnvCharacterSetConverter::AutoDetectCharacterSetL that the plug-in DLL is actually implementing a function of this signature and not the old first reserved function. The meanings of the remaining parameters is the same as the meanings of the corresponding parameters in CCnvCharacterSetConverter::AutoDetectCharacterSetL.

CCnvCharacterSetConverter::AutoDetectCharacterSetL

Internally AutoDetectCharacterSetL loops through the available aArrayOfCharacterSetsAvailable and checks whether that character set is a plug-in supported one. If it is, then it loads the corresponding plug-in DLL and calls the DLL's exported function at ordinal 4 (checking aSetToTrue to verify that the plug-in DLL is actually implementing this function as IsInThisCharacterSetL - it may still be implementing it as Reserved_1 which is perfectly legitimate). IsInThisCharacterSetL  then sets a confidence level for that character set for the sample. If the character set is not one that is supported by the plug-in libraries, it then loops through the list of character sets that are supported internally by Charconv and call their character specific functions, e.g. IsCharacterSetAscii, IsCharacterSetSMS7Bit etc. These functions also set a confidence level for that character set for the sample. Once it has looped through the entire array of aArrayOfCharacterSetsAvailable the character set with the highest confidence level is returned as the character set encoding the sample text is encoded in. 

If the maximum confidence-level encountered in this loop is less than some threshold (say, 25), then CCnvCharacterSetConverter::AutoDetectCharacterSetL resorts to attempting to convert aSample to each of the character sets in aArrayOfCharacterSetsAvailable. The character set that generates fewest 0xfffd Unicode "REPLACEMENT CHARACTER"-s wins, with aConfidenceLevel being set according to some formula based on the difference between the number of 0xfffd characters generated by that character set and its nearest rival. If the maximum confidence level encountered (in the loop mentioned above) is shared by more than one character set, it will make the best guess possible.

If there are more that one character set encoding that have the same confidence level, CCnvCharacterSetConverter::AutoDetectCharacterSetL will convert the sample text aSample  in each character set encoding with the same confidence level and the one with the least number of Unicode replacement characters wins. 

IsInThisCharacterSetL

Each of the character sets supported by the plug-ins have their own implementation of IsInThisCharacterSetL. This is achieved by using signature sequences associated to a language/  encoding which is the same technique employed in virus detection. One such example would be if the encoding was a Modal Encoding - where escape characters/sequences or special characters are used to switch between different character modes 1-byte, 2-byte or other character set. For e.g. ISO-2022JP is a modal encoding. The following are the escape sequences and their meaning. 

Hex values of keys that make up the escape sequence. ISO2022JP is a multi-byte encoding. The range for the 1st byte and the 2nd byte are the same, i.e. between the hex value 0x21 & 0x7E.  

0x1B 0x28 0x42  - ASCII escape sequence. 
0x1B 0x28 0x42 - JIS-Roman escape sequence. 
0x1B 0x24 0x40 - JIS C 6226-1978. 
… etc. 

The escape sequences tell what character the text following the escape sequence is encoded in. 

So in AutoDetectCharacterSetL when ISO2022 is the character set in aArrayOfCharacterSetsAvailable, the IsInThisCharacterSetL implementation for the ISO2022JP’s plug-in is called. In that function, it is checked that all the characters that make up the sample text are in range, if it is not in range, i.e. if there exist characters whose values are greater than 0x7e then the sample text is not ISO2022JP. So we can give it a confidence level of 0. Then it looks for the escape sequences. The presence of the escape sequences that are unique (not completely unique though. For example ISO2022JP1 is exactly the same as ISO2022JP + an extra escape sequence)  to this character set increases the possibility of the sample text being encoded in this character set encoding. 

EUCJP encoding uses shift characters. 0x8E and 0x8F are the single shift characters. The range of the encoded characters + the single shift character tells whether that byte is part of a 2-byte character. Here are the ranges of EUCJP

0x00 - 0x7F   
[0xA1 - 0xFE] [0xA1 - 0xFE]  2 byte character with the range of each byte shown brackets. 
0x8E [0xA0 - 0xDF]  single Shift Character 0x8E 
0x80 [0xA1 - 0xFE] [0xA1 - 0xFE] 

The sample text provided affects the outcome of the function. If the sample text is very small, there might be only one escape sequence which is not enough to give a 100% accurate answer. In order to gain more knowledge of different Character set encoding, read  chapter 4 of the following 

“CJKV Information Processing” - by Ken Lunde. 

Sample Code

Here are a few working sample of Autodetection. 

As mentioned earlier the aArrayOfCharacterSetsAvailable parameter is included for efficiency - CCnvCharacterSetConverter::AutoDetectCharacterSetL could itself find the list of character-sets available but this is a very time-consuming task and the client may already have this information to hand. If the client does not have that information then the following creates an array of character sets

RFs fileServerSession;
CleanupClosePushL(fileServerSession);
User::LeaveIfError(fileServerSession.Connect());
CCnvCharacterSetConverter* characterSetConverter=CCnvCharacterSetConverter::NewLC();
CArrayFix<CCnvCharacterSetConverter::SCharacterSet>* arrayOfCharacterSetsAvailable=characterSetConverter-> CreateArrayOfCharacterSetsAvailableLC(fileServerSession);

then the AutoDetectCharacterSetL function can be used

_LIT8(KASCII, “The result I am expecting is that this is recognised as ASCII!");
TInt Confidence = 0;
TUint Character = 0; 
CCnvCharacterSetConverter::AutoDetectCharacterSetL (Confidence, Character, *arrayOfCharacterSetsAvailable, KEUP);

The sample text is ASCII encoded plain text. Confidence and Character are where the confidence level and the winning Character set are returned.