Skip to main content
GameDev.net gamedev.net
🔒 Locked

Unicode best practices

Started by ApochPiQ Aug 28, 2008 at 4:46 PM 29 replies 10.1k views
Original Post
ApochPiQ
ApochPiQ
I'm doing my best to be Unicode compliant, but a lot of libaries still expect ANSI/UTF-8 and I'm using std::wstring internally. Question is, what's the best way to widen and narrow strings? This is the code I'm using currently:
std::wstring widen(const std::string& str)
{
	std::wstring ret;
	ret.reserve(str.size());
	for(std::string::const_iterator iter = str.begin(); iter != str.end(); ++iter)
		ret += std::use_facet<std::ctype<wchar_t> >(std::wcout.getloc()).widen(*iter);

	return ret;
}

std::string narrow(const std::wstring& str)
{
	// EVIL HACK
	std::string ret;
	ret.reserve(str.size());
	for(std::wstring::const_iterator iter = str.begin(); iter != str.end(); ++iter)
		ret += static_cast<char>(*iter);

	return ret;
}
The widen function works beautifully, but the narrow function basically just prays to God that the string is in UTF-8 and therefore the typecast will output meaningful data instead of gibberish (not to mention overflow problems). So if anyone knows a better way to do either of these tasks, I'd love to learn it!
SiCrane
SiCrane
Well, the best way would be to use a library like ICU that keeps up-to-date with the current Unicode standard (and is actually guaranteed to be using Unicode, which you aren't guaranteed if using C++ standard library components). If you don't to choose to use a third party library, you should choose to use string oriented widening and narrowing functions. From the standard C library you've got wcstombs() and mbstowcs(). From the standard C++ library the std::ctype<> facet contains both narrow() and widen() member functions that work on strings (unfortunately the char * or wchar_t * sense of strings rather than string objects). It's almost never a good idea to use the single character versions since many narrowing operations on a single character can actually result in multiple characters. For example, the German ß when narrowed results in the string "ss".
Bregma
Bregma
(1) UTF-8 is Unicode.

(2) You can't just widen and narrow characters. You need to convert between encodings. For that, you need a codec. IBM's ICU is good for Unicode (you can use it to convert between UTF-8 and UCS-4, for example, or to the non-standard UTF-16 used on Windows).

Pretty much everything ICU does should be encapsulated in C++ locale functionality, assuming you have the right locales available in your runtime (ICU also supplies those).

Here's a basic rule of thumb for handling character encodings portably in C++: use wide characters and stick to UCS-4 (or Microsoft's UNICODE) internally, and stick to byte-oriented chars and UTF-8 externally if possible. Make sure the codecvt() facet of your current locale will convert to/from your external representation and your internal representation, stick to standard I/O for all your I/O needs, and then you never need to worry.
Stephen M. Webb
Professional Free Software Developer
ApochPiQ
ApochPiQ
Thanks, I'll have a look at ICU; it sounds like it'll solve my problem.

Quote:
UTF-8 is Unicode.


Of course - but if the wide string is in any other encoding, and has characters that can't be represented in UTF-8, my current narrow will produce garbage... won't it?
Bregma
Bregma
Quote:
Original post by ApochPiQ
Quote:
UTF-8 is Unicode.

Of course - but if the wide string is in any other encoding, and has characters that can't be represented in UTF-8, my current narrow will produce garbage... won't it?

I would like to see what non-Unicode encoding you might encounter would not also be representable in Unicode.

Your current narrow() is likely to produce garbage regardless, since it does no conversion between codesets. Even you widen() isn't right, since the standard ctype facet does not support multibyte codesets.

Like I said, stick to using a wide-character Unicode (preferrably UCS-4, Microsoft's UNICODE if you must) internally. You have control over what you do internally, so be consistent and predicatable. Rely on your codecs (in particular, the codecvt() facet) to convert from your known internal codeset into whatever external codeset you need. If you need on-the-fly conversion without using standard I/O, you need to use mbstowcs() and friends (a C API) or else ICU.
Stephen M. Webb
Professional Free Software Developer
ApochPiQ
ApochPiQ
I'm 100% wstring/wchar_t* internally, using the UNICODE option in VS2005. The problem lies with external libraries that expect narrow strings; I've got to communicate with them somehow [wink]

ICU looks like way too much of a dependency (it's 6 times the size of my program, for the love of Bob) so I'll settle for mbstowcs and company.


[edit] (Actually - about your last - what would the "standard I/O" method be?)
Jan Wassenberg
Jan Wassenberg
You might want to check out ConvertUTF.h from unicode.org - that's a fairly simple library for converting between UTF-8/UCS-2/UCS-4.
Note that sizeof(wchar_t) (and thus std::wstring) differs between Visual C++ and gcc, which is annoying.
E8 17 00 42 CE DC D2 DC E4 EA C4 40 CA DA C2 D8 CC 40 CA D0 E8 40E0 CA CA 96 5B B0 16 50 D7 D4 02 B2 02 86 E2 CD 21 58 48 79 F2 C3
ApochPiQ
ApochPiQ
Right... here's what I'm now using, adapted from here:

//// Convert a narrow string to a wide string//std::wstring widen(const std::string& str){	const char* cstr = str.c_str();	size_t len = str.length() + 1;	size_t reqsize = 0;	mbstowcs_s(&reqsize, NULL, 0, cstr, len);	if(!reqsize)		throw std::exception("Failed to widen string");	std::vector<wchar_t> buffer(reqsize, 0);	mbstowcs_s(NULL, &buffer[0], len, cstr, len);	return std::wstring(buffer.begin(), buffer.end());}//// Convert a wide string to a narrow string//std::string narrow(const std::wstring& str){	const wchar_t* cstr = str.c_str();	size_t len = str.length() + 1;	size_t reqsize = 0;	wcstombs_s(&reqsize, NULL, 0, cstr, len);	if(!reqsize)		throw std::exception("Failed to narrow string");	std::vector<char> buffer(reqsize, 0);	wcstombs_s(NULL, &buffer[0], len, cstr, len);	return std::string(buffer.begin(), buffer.end());}


Anyone note any problems?
Bregma
Bregma
Quote:
Original post by ApochPiQ
(Actually - about your last - what would the "standard I/O" method be?)


IOStreams? If you have a wstring in, say, UTF-16, and you need a multibyte string in, say, ISO-8859-7 or maybe UTF-8, use a std::ostringstream imbued with an appropriate locale, stream out your wide string, retrieve your narrow string. Also works vice-versa.
  #include <locale>  #include <sstream>  std::string make_narrow(std::wstring ws)  {    std::ostringstream ostr;    ostr.imbue(std::locale("en_US.utf8"));    ostr << ws;    return ostr.str();  }


The trick is in finding the right locale.

You could also use the std::codecvt facet directly if you're a sucker for pain, since it has less overhead, but it sure suffers from clarity. The word bletcherous comes to mind. You still need to find the right locale.
Stephen M. Webb
Professional Free Software Developer
ApochPiQ
ApochPiQ
Hmm... that's definitely cleaner than the C API method, but it leaves the major question of selecting the correct locale, as you noted. Problem is I have no experience whatsoever working with the C++ locale stuff, so I'd have no clue where to even begin with ensuring that the conversion locale was correct on a given system.
Bregma
Bregma
Quote:
Original post by ApochPiQ
Anyone note any problems?

You will need to make sure you set your global C runtime locale using setlocale(). The examples MS give don't do that so they use the default locale, which may or may not do what you want (if the default is to use UTF-8 and your third-party library expects, say, ISO 8859-1, then you're in for a big, and rather unpleasant, surprise -- it may also be the other way around).

Again, you will need to make sure you have the right locale.
Stephen M. Webb
Professional Free Software Developer
Washu
Washu
Unfortunately, unicode support in the C++ standard library is...mostly a half assed effort.

As SiCrane said, if you really want to support unicode, you're going to have to go through a third party library. There are also a lot of quirks in the wide implementation of the standard library that actually can make things like reading and writing files potentially broken.

Even the localization functionality, while a decent effort, is also somewhat limited and broken in its usage. (Sorry, but std::use_facet >(loc).put(std::ostreambuf_iterator(ss), ss, ' ', value); is not something one should ever have to type, not to mention the behaviors aren't even well defined with regards to localization and unicode.)
In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ApochPiQ
ApochPiQ
So it sounds like this may end up being more trouble than it's worth.

Any opinions on whether or not to just scrap the effort and go with std::string everywhere?
frob
frob
Quote:
Original post by ApochPiQ
So it sounds like this may end up being more trouble than it's worth.

Any opinions on whether or not to just scrap the effort and go with std::string everywhere?

What's it for?

If it is internal, or if you can honestly expect that it will only be used in UTF-8 or similar environment, then go with std::string.

If somebody requires Unicode, they're already used to bajillions of other programs not supporting it, so what's one more?

If it is for external release, or expected to be used globally, then you should invest the time to use an external library (or another language that does a better job with Unicode).
gdunbar
gdunbar
I'm not an expert on localization or anything, but when I worked at Microsoft we used WideCharToMultiByte and MultiByteToWideChar extensively, in products that were localized into dozens of languages. They are Win32 (C friendly) calls, but wrapping them up to make them std::wstring friendly shouldn't be too hard.

All moot if you are trying to be cross-platform, of course.

Good luck,
Geoff
ApochPiQ
ApochPiQ
This is for Fugue, the reference implementation and VM for Epoch, my general-purpose programming language. (Journal has a bunch of details if anyone cares.)

I was really hoping to support Unicode out of the box with the language, precisely because so many programs do not; but if the cost is rewriting half my string handling to use Yet Another String Class, I'm having second thoughts.

Which is not to say it isn't worth the effort... I'm just rather ambivalent at the moment.
Washu
Washu
Quote:
Original post by ApochPiQ
This is for Fugue, the reference implementation and VM for Epoch, my general-purpose programming language. (Journal has a bunch of details if anyone cares.)

I was really hoping to support Unicode out of the box with the language, precisely because so many programs do not; but if the cost is rewriting half my string handling to use Yet Another String Class, I'm having second thoughts.

Which is not to say it isn't worth the effort... I'm just rather ambivalent at the moment.


It ends up be a lot more than "Yet another string class." you end up having to think in fundamentally different terms. No longer are you dealing with just different characters, but glyphs that compose characters. The principles and patterns you use change as well, since the behavior of them is quite a bit different.

Some languages, for instance, the addition of a character requires backtracking and changing previous characters and glyphs, and possibly even inserting new glyphs. These are, hopefully, handled by your unicode library, however that does mean that you can't end up assuming some things like you used to (such as that the length of a string + new character is len(string) + 1). Japanese is a prime example, by mixing Hiragana, Kanji, and Katakana in a single document (quite common) you often will end up writing in Hirigana and in the process certain sequences of characters will be simplified to their Kanji equivelent. This can end up eliminating whole blocks of characters, and replacing still others.
In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
SiCrane
SiCrane
Unicode in programming languages can get complicated. What kind of Unicode support are you looking at here? Unicode identifiers? Unicode string constants? Unicode string objects?

Using Unicode identifiers opens up a special kind of pain for lexical analysis especially if you follow the Unicode consortium's guidelines on Unicode identifiers. Basically the Unicode guidelines suggest that identifiers take the form {XID_START}{XID_CONTINUE}* Most languages would extend the XID_START and XID_CONTINUE character classes with additional code points like _. Of course when you do that you have the issue of telling the difference between the identifier pathfinding and the identifier pathfinding (in case you can't tell, the second identifier uses the ligature fi). So proper handling requires normalization, generally with NFKC. If you do so, then you'll need to go outside the standard library for this support.

For Unicode string objects in your language, the situation may be better. However, you're using MSVC so wchar_t is 16-bits, which means that you've got the wonderful issue that single wchar_t units could be halves of surrogate pairs, so the concept of [] indexing string objects takes a hard whack even in the simple read-only string situation. It gets better if you use iterators rather than indexing for string operations. However, if you try code point assignment into arbitrary points into your string you'll find that writing over a BMP character with a surrogate pair or vice versa requires moving all the wchar_ts after the write point. Choices here include having immutable string objects, switching up to a 32-bit integer type for strings or doing this kind of memory movement whenever needed.
Spoonbender
Spoonbender
Quote:
Original post by ApochPiQ
This is for Fugue, the reference implementation and VM for Epoch, my general-purpose programming language. (Journal has a bunch of details if anyone cares.)

I was really hoping to support Unicode out of the box with the language, precisely because so many programs do not; but if the cost is rewriting half my string handling to use Yet Another String Class, I'm having second thoughts.

Which is not to say it isn't worth the effort... I'm just rather ambivalent at the moment.


I don't blame you, but I'd say a modern language that doesn't have out-of-the-box unicode support is pretty much useless. I mean, as you're finding out, it's not something that can be easily bolted onto a language afterwards. If you're serious about this language being useful, I'd say unicode is a must.

In any case, I've wondered about the same thing. It does seem awfully hard to get proper unicode support in C++.

ICU just looks amazingly huge and complicated and, well, the Java heritage visible in the API is kinda off-putting. You'd think someone somewhere had made a more STL-compatible API. (Or maybe I've just been looking at the wrong parts of ICU?)


By the way, I know C++0x is adding UTF8/16/32 character types, but does anyone know if that means actual proper unicode support throughout, with all the standard library changes that'd entail?
SiCrane
SiCrane
Quote:
Original post by Spoonbender
ICU just looks amazingly huge and complicated and, well, the Java heritage visible in the API is kinda off-putting. You'd think someone somewhere had made a more STL-compatible API. (Or maybe I've just been looking at the wrong parts of ICU?)

ICU's C++ bindings are very non-idiomatic C++. Basically, ICU is designed to operate in environments where things like C++ exceptions aren't happy, so the ICU C++ API reflects this heritage. I personally don't see the point, since ICU has a perfectly functional C API for those environments.


Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.