UTF-8 char* strings

Quote:

Original post by mackron [are there] any platforms that assume UTF-8 encoding for their str*() family of C library functions? That way I can use those routines instead of writing my own.

I'm afraid (AFAIK) the standard C library has no such thing.
The good news is, that you don't have to write your own; unicode.org already has!

If you're still interested in writing your own, there is this article I came across recently.

The standard C library is... kind of... older than Unicode (and certainly older than the popular adaptation of Unicode), so no, it's not going to work that way. The signatures for these functions are defined by the standard to accept and return char*, and the definition of e.g. strlen() requires it to count all the chars until a '\0', which rules out UTF-8 (since it's a variable-length encoding).

There are certainly lots of existing code snippets for doing the necessary translation work.

Thanks guys.

Quote:

The good news is, that you don't have to write your own; unicode.org already has!

Yeah I saw that, and I've based all of my conversion functions off of that code.

Quote:

The standard C library is... kind of... older than Unicode (and certainly older than the popular adaptation of Unicode), so no, it's not going to work that way. The signatures for these functions are defined by the standard to accept and return char*, and the definition of e.g. strlen() requires it to count all the chars until a '\0', which rules out UTF-8 (since it's a variable-length encoding).

I always thought strlen() is supposed to calculate the number of bytes and not necessarily the number of characters. I think I might have read that from MSDN...

So will the str*() family of functions only work with ASCII strings?

Thanks again.

Quote:

Original post by mackron I always thought strlen() is supposed to calculate the number of bytes and not necessarily the number of characters. I think I might have read that from MSDN...

That is correct, str* works on bytes not characters.

Quote:

Original post by mackron So will the str*() family of functions only work with ASCII strings?

With the above in mind, str* will still "work" with UTF-8 strings, depending on what it is you actually expect it to do (i.e. it doesn't "work" if you expect it to count characters, but it does "work" if you expect it count bytes).

By the way, I should just point out that even wcslen and friends doesn't really count "characters" either. It counts UTF-16 code points. After all, "é" is one character, but wcslen("é") would return 2, because the string is actually .

Quote:

Original post by Codeka By the way, I should just point out that even wcslen and friends doesn't really count "characters" either. It counts UTF-16 code points. After all, "é" is one character, but wcslen("é") would return 2, because the string is actually .

The funny thing about Unicode is that there are several ways to get certain characters: é.

Quote:

Original post by ToohrVyk The funny thing about Unicode is that there are several ways to get certain characters: é.

That was kind of my point [smile]