### Multibyte and Unicode Support

The Carbide preprocessor fully supports the use of multibyte and Unicode-formatted source files.

Source and header files may be encoded in any text encoding the operating system recognizes. (Unicode text decoding support is implemented using native OS services in Win32 and conv() on UNIX systems.)

As per ISO C++98 and C99, universal characters may be used in any string or character literal, identifiers (macros, variables, functions, methods, etc.), and comments. These characters are derived either from multibyte sequences in the source text, by virtue of the source file being encoded in UCS-2 or UCS-4, or by use of the \uXXXX or \UXXXXXXXX escape sequences.

#### Wide String Literals

Wide string literals in the form L"xxx" and wide characters in the form L’xxx’ are interpreted in the context of the source text encoding.

const wchar_t *str = L"Meine Katze ißt Mäuse nachts.";

Multibyte character sequences that appear in strings are internally converted to their Unicode equivalents until the C/C++ token for the string is generated. At that time, if the string literal is a narrow string (i.e. using char), the original multibyte character sequence are emitted. If the string is a wide string (using wchar_t), the Unicode characters translated from the multibyte sequence are emitted. If wchar_t is 16 bits and a character is truncated to 16 bits, a warning is reported. (See ISO C99, §6.4.5.5 for the specification of this behavior.)

In the event that you want translated – and usually truncated – Unicode characters in narrow string literals, enable #pragma multibyteaware_preserve_ literals off.

#### Source Encoding

The compiler uses the Source encoding option in to control how it detects the source file encoding. The compiler recognizes the following source encoding options:

- ASCII–no detection is done, the high-ASCII characters are not interpreted, and wide character strings are merely zero-extensions of the individual bytes in the string.
- Autodetect–the compiler attempts to tell whether byte-encoded source is encoded in UTF-8, Shift-JIS, EUC-JP, ISO-2022-JP, or whichever encoding format the operating system considers the default. This option degrades compile speeds slightly due to the extra scanning.
- System–the compiler uses the system locale without scanning the source.
- UTF-8, Shift-JIS, EUC-JP, ISO-2022-JP– self-explanatory.

NOTE Currently, the compiler ignores the mapping in some Japanese character sets of 0x5c (ASCII ‘\’) to Yen (\u00A5) because the C++98 and C99 standards say that the ASCII character set must be mapped one-to-one with any multibyte encoding. 0x5C is always interpreted as ‘\’ (except inside multibyte character sequences).

NOTE The ISO-2022-JP and EUC-JP encoding currently only recognize characters defined by JIS X 0208-1990 (i.e., the escapes ESC $ @, ESC $ B for ISO-2022-JP and two-byte sequences in EUC-JP). The additional characters in JIS X 0212-1990 are not yet recognized.

No matter what the Source encoding setting, the compiler always detects UTF-16{BE,LE} or UCS-4{BE,LE} source through a statistical character scan for NULs.

NOTE Currently, only the command-line tools, not the IDE, can properly handle sources in Unicode format (UTF-16, UCS-2, UCS-4).

Alternately, individual source files can identify which source text encoding is present using #pragma text_encoding. The format is:
#pragma text_encoding("name" | unknown | reset [, global])

Where name is an IANA or MIME encoding name or an OS-specific string.

The compiler recognizes these names and maps them to its internal decoders:

system US-ASCII ASCII ANSI_X3.4-1968 ANSI_X3.4-1968
ANSI_X3.4 UTF-8 UTF8 ISO-2022-JP CSISO2022JP ISO2022JP
CSSHIFTJIS SHIFT-JIS SJIS EUC-JP EUCJP
UCS-2 UCS-2BE UCS-2LE UCS2 UCS2BE UCS2LE UTF-16 U
TF-16BE UTF-16LE UTF16 UTF16BE UTF16LE UCS-4 UCS-4BE
UCS-4LE UCS4 UCS4BE UCS4LE 10646-1:1993 ISO-10646-1
ISO-10646 unicode

NOTE UCS-2 is always interpreted as UTF-16; i.e. surrogate character pairs are used to select characters through plane 16.

This #pragma may be used several times in one file (probably unlikely usage). The compiler expects the pragma to be encoded in the “current” text encoding, through the end of line.

By default, #pragma text_encoding is only effective through the end of file. To affect the default text encoding assumed for the current and all subsequent files, supply the “global” modifier.

#### Other Support

In addition, note the following:

- Specify Universal character names (i.e. Unicode code points) with the \uXXXX or \UXXXXXXXX escape sequences:

> wchar_t florette = L’\u273f’;
> int \u30AD\u30B3\u30DE\u30A6 = soy_sauce();

- You can use \uXXXX or \UXXXXXXXX regardless of the actual size of wchar_t (use of the escape does not impose a character width on the literal).
- Preprocessed text is always emitted in ASCII. Wide characters are emitted in the \uXXXX or \UXXXXXXXX format.

> extern string *Wörter[];
> -->
> extern string *W\u00F6rter[];

- Identifiers using characters not representable in ASCII are emitted in object code with the \uXXXX or \UXXXXXXXX escape sequences. If an object format does not support the ‘\\’ character, the encoding may be modified on a target-specific basis.

Also see “Character Constants as Integer Values” for information on creating a character constant consisting of more than one character (not to be confused with this topic).

For more information on Unicode, see www.unicode.org.