string::length and unicode

UTF-8 stores certain characters using two or even three bytes. Since std::string stores byte-size characters, you can store UTF-8 text safely, but you will not be able to manipulate the characters themselves.

Your options are to use specific UTF-8 string manipulation functions, or to use std::wstring and UTF-16.

Even UTF-16 won't do it. In the fully generaly case with surrogate pairs, combining diacritics, and variation selectors a "character" can be pretty much as long as you want it to be, depending on your definition of a character.

My main question is why exactly do you care? There aren't very many practical uses for true character counts.

Quote:

Original post by Anon Mike There aren't very many practical uses for true character counts.

Line-wrapping for monospaced fonts?

Would this problem be solved by UTF-32? As I understand it, UTF-32 has no surrogate pairs.

It's a little more complicated than that. Even with 4 bytes per character, multiple code points can be used to create a single grapheme. For example, a-umlaut (ä) can be described both as a single code point U+00E4 or a sequence of two code points U+0061 U+0308 (lower case a followed by a combining diaeresis). If you just use code points to count then you'd get the wrong number of characters for things like line-wrapping in monospaced fonts if you having combining characters.

Tell us what you are trying to accomplish then we can suggest a better way of doing things.

Quote:

Original post by Nitage Tell us what you are trying to accomplish then we can suggest a better way of doing things.

Yes, of course. Line wrapping is close.
I am writing text based interface (ncurses, cdk) and I want to display a table.
Like this (I do not now I to get fixed font size, but you get the Idea):

Entry1 | Entry2 | Entry3
___________________________________
A | B | C
A2 | B2 | C2
...

For this, I wrote a function:

string fix_size(string s,int size)

Returning a string of fixed size (by appending spaces or removing characters).

This is actually non-trivial for generic Unicode text, as SiCrane pointed out (because one "character" can be made up of multiple Unicode code points). This is especially true if you want to handle something like Vietnamese, which uses a lot of combining characters.

One solution is ICU which has methods specifically for iterating over strings. Though ICU might be a bit heavy-weight for you. If mostly-European languages is all you care about, you can probably achieve a "good enough" solution knowing that the UTF-8 encoding ensures that the first byte of a codepoint has the high bit unset, and all other bytes have the high bit set. So you can work out character boundaries that way. That should cover you for 90% of European languages (and probably quite a bit of other languages as well), I'd say.

That would depend on the input source, I think, as that still only gives you code point boundaries. The a-umlaut example is something that can occur in, for example, German Unicode text, so you still need to watch out for combining characters. I suppose you could try running the text through NFC or NFKC to combine code points before doing a count, but again that requires using a library like ICU.

Are there any codepoints that are not combinable? If not, then wouldn't it be possible on Windows to use the FoldStringW call and then either
1) use wstrlen
or
2) convert the resultant wchar_t array into a std::wstring and invoke its size method?

Quote:

Are there any codepoints that are not combinable?

The reasons combinations exist is because hard-coding all combinations would take too much space in Unicode.

So no, normalizing to C (the compact form) is not a solution.
But it's still fairly simple. For every code point, add 1 to your counter unless the code point is a combining character.

So, when dealing with UTF-8, you first need to iterate the code points within the encoded Unicode string. I find doing that with an iterator adaptor practical.

Quote:

Original post by Khaos Dragon Are there any codepoints that are not combinable?

If you're asking if there are any legitimate sequence of code points that result in a single grapheme that lacks a single code point identity in Unicode, then the answer is yes. In particular graphemes consisting of letters with accents both above and below the letter tend not to have single code point values. Though there are exceptions like ṩ (U+1E69).

Quote:

Original post by SiCrane Quote: Original post by Khaos Dragon Are there any codepoints that are not combinable? If you're asking if there are any legitimate sequence of code points that result in a single grapheme that lacks a single code point identity in Unicode, then the answer is yes. In particular graphemes consisting of letters with accents both above and below the letter tend not to have single code point values. Though there are exceptions like ṩ (U+1E69).

Yes that is what I was asking. So how do libraries like freetype font handle this? Does freetype handle fully compositing multiple code points into a grapheme or does the user need to composite the glyphs freetype returns from each code point himself? Excuse me if this treads outside the topic of the thread, but I am not really asking for a code sample, just trying to get a stronger conception of what this entails since I have just realized my game that I thought was unicode conformant is suddenly not.

One additional thought: there should be a reference string which utilizes multiple combining codepoints from a wide set of character sets in order for a text system to test its unicode conformity. Something like the acidtest for text handlers.

Quote:

Original post by loufoque But it's still fairly simple. For every code point, add 1 to your counter unless the code point is a combining character.

It's more complicated than that. When you encounter a combining character you also need to check that your text renderer will recognize the combination as a single width character. For example if you throw U+0061 U+0308 in a text file and open it with Notepad (on one of the Unicode aware Notepad versions anyways), it will display as a single width character. Put in U+0061 U+0308 U+0308 U+0308 and it will take up the width of four characters (a followed by three sets of high dot pairs).

This is not handled by Freetype, but rather by Pango, Uniscribe or Graphite.

I'm not an expert on unicode, but based on the comments made by other posters so far, it sounds like there are certain characters that are always used as 'modifiers' and other characters that are used as actual base symbols. If that is the case, then you could simply write a function that counts the number of symbols which are modifiers, and subtract that number from the string length.

Quote:

Original post by yahastu I'm not an expert on unicode, but based on the comments made by other posters so far, it sounds like there are certain characters that are always used as 'modifiers' and other characters that are used as actual base symbols. If that is the case, then you could simply write a function that counts the number of symbols which are modifiers, and subtract that number from the string length.

Quote:

Original post by SiCrane It's more complicated than that. When you encounter a combining character you also need to check that your text renderer will recognize the combination as a single width character. For example if you throw U+0061 U+0308 in a text file and open it with Notepad (on one of the Unicode aware Notepad versions anyways), it will display as a single width character. Put in U+0061 U+0308 U+0308 U+0308 and it will take up the width of four characters (a followed by three sets of high dot pairs).

If memory serves me right, an extended utf-8 char is a number of bytes with bit 7 set and the last one has it clear. So an easy way to find the length is to go through the string and count all the ones less than or equal to 127.

1) Your memory is flawed. That won't properly count the number of code points. That would only count ASCII characters and completely miss any code points above 127, as your condition would completely miss any characters that require multiple bytes to encode.
2) If you had bothered reading the thread, you would have noticed that whole discussion about combining characters.