Skip to main content
gamedev.net
🔒 Locked

Sorting unicode strings

Started by roos 3 replies 1.1k views
Original Post
roos
roos
Hi, I'm using unicode strings (basic_string<uint16>) for my game so it can be localized to any language. I was wondering, is there any good way to compare two unicode strings to see which one is alphabetically larger? With regular strings, I could just use strcmp() and check if it's greater than zero or whatever. With unicode strings I'm not sure... If I knew how to compare two characters to see which one is bigger, then writing the compare function is easy enough... Is it okay to just compare the values of the characters to see which one is bigger? In English I think that would work, because afaik the unicode value for "d" is less than the value for "z" for example. In Japanese I don't know how to go about it, since you might have two different Kanji with different unicode values, but the same phoenetic sound. Hmm, I'm going to spend a little more time looking into this, but if anyone might know offhand if there's a proper way to compare unicode strings, I would really appreciate it if you could give me a hint! Thanks, roos
Fruny
Fruny
Quote:
Original post by roos
I'm using unicode strings (basic_string<uint16>) for my game so it can be localized to any language.


Just using a wide string type doesn't magically make them Unicode strings, you need to use a Unicode locale too. This is what you need to start doing research on.

Quote:
I was wondering, is there any good way to compare two unicode strings to see which one is alphabetically larger?


Use the std::collate facet of the appropriate locale.

Quote:
Is it okay to just compare the values of the characters to see which one is bigger?


It depends on whether you want the comparison to be meaningful (dictionary order) or just be able to sort strings in an arbitrary order e.g. to do a binary search.

Quote:
In English I think that would work, because afaik the unicode value for "d" is less than the value for "z" for example.


And yet you still have to deal with the fact that the value for 'Z' is less than the value for 'a'.

Quote:
In Japanese I don't know how to go about it, since you might have two different Kanji with different unicode values, but the same phoenetic sound.


unicode.org
Collation
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." — Brian W. Kernighan
SiCrane
SiCrane
Ah the joy of collation. Alphabetical sorts don't work well with unicode strings. Even limited to the realm of just a-z, you have languages like Lithuanian which put letters in different order than the ASCII ordering. On the plus side the Unicode organization has a document which describes how to do collation on Unicode strings: UTS #10: Unicode Collation Algorithm. On the minus side it's somewhat annoying to implement.

If you want something faster, but still handles many of the annoying corner cases, you may consider first normalizing the string and then doing a lexigraphic ordering based on the numeric value of code points. Depending on your appication different choices many be made on the normalization. NFKC or NFKD would give you less trouble with things like ligatures, but if you may conisder NFC or NFD if rendering information is important.
roos
roos
Wow, I see it's a lot more complicated than I thought! Well, that definitely helps a lot because now I have some links and some terms I didn't know about like "collation". Thanks a ton Fruny and SiCrane for your help, I appreciate it very much.
SiCrane
SiCrane
And this is one of the reasons that I recommended using ICU in your last thread about Unicode strings. ICU handles things like collation and normalization for you.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.