UNICODE and string confusion...

It's really simple.

Use Unicode only if you intend to make your program multi-langual or allow support for multiple
languages in the future. If your program will only use English, now and forever, then don't use Unicode.

maybe my fault...

I WILL USE UNICODE - period.

should I base it on ?? or windows TCHAR?

I noticed that you had to define
#define _UNICODE // for windows
#define UNICODE // for

any comment regarding using those two different libraries?
any comment on using little or big endian?

Both are windows concepts. Neither are portable to other platforms. The TChar stuff is windows specific. THe theorey is portable but not the types.

wchar_t is 16 bits on windows and 32 on OS X for instance, allthough gcc will let you compile with the whcar-short flag, but then you can't use the C++stdlib on OS X as tis compiled to use wchar_t as 4 bytes.


If all you need is windows support then use the TCHAR defs. 10s of thousands of programs have been written usign them to successfully support unicode on the PC.

Cheers
Chris

i would use the Win32 functions. they support UTF-16. so use unsigned short instead of char. you could easily write your own string class if you use the Win32 functions. just remember to use reference counting so that you can return a string from a function.

ex:
string Function0()
{
string ret = L"pizza";
return ret;
}

TCHAR is for apps that want to do both Unicode and non-Unicode depending on how they're compiled. TCHAR is really just an alias for char or WCHAR depending on your settings.

If you want to be Unicode on Win32 then use WCHAR. WCHAR is a 16-bit little-endian type. You still need to define UNICODE and _UNICODE so that you end up calling Unicode versions of the API. In case you don't know, for any Win32 API that takes character data, say TextOut, there are really two versions - TextOutA and TextOutW. TextOut itself is just a #define that selects which of the 'A' and 'W' versions to use.

When using Unicode all of the standard C/C++ runtime functions like sprintf have Unicode equivalents. swprintf in this case. MSDN will tell you what the unicode version is and it will usually be basically the same name except with a 'w' in it someplace.

You can't really manipulate UTF-8 directly in Win32, pretty much none of the API's will take it as it's not a real code page in the Windows view of things. You can convert UTF8 data using MultiByteToWideString().

Quote:

Original post by raydog It's really simple. If your program will only use English, now and forever, then don't use Unicode.

How do you know someone who doesn't have an english version of windows won't use your program?

Make your programs as simple as possible and no simpler.

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

Its important to note that windows uses UTF16 and this won't handle all unicode characters. At one time it was thought it would:). This really isn't microsofts fault as they did their unicode work back in 91 way before Mac and linux were thinking of this stuff.

Mac and linux use ucs4, (UTF32), This encoding does handle all possible unicode characters.

With UTF16 you'll need to handle surggates if you want to handle all characters.

CHeers
Chris

You don't need to define a custom string type, the standard library string is a class template, std::string is just a type alias for the the real type std::basic_string, 2 of it's type parameters are the character type and character traits, there is also a type alias of basic_string for the wide character type wchar_t.

You can also do localization/internationalization in the standard IOstreams library, look up locales, facets, character traits in particular the facet std::std::codecvt/codecvt_byname preforms code conversation to and from the internal and external respenstation specified by the character encoding schemes.

You need to decide what your going to use for your internal representation, character type and character encoding scheme, you need to know what character type and character encoding scheme is used for the external repesentation.

Quote:

Original post by petewood How do you know someone who doesn't have an english version of windows won't use your program?

I'm not saying someone from Japan, for example, can't use my English-only program on their version of Windows.
They can. The text strings just won't be in Japanese.

What I'm saying is that I'm not going to waste any time and translate every English string into
10 different languages. If I was selling my program internationally, then perhaps I would, but
I'm not, so it really doesn't matter.

Quote:

Original post by raydog The text strings just won't be in Japanese.

What will they be?

Are you sure?

Quote:

Original post by petewood Quote: Original post by raydog The text strings just won't be in Japanese. What will they be? Are you sure?

English, of course. The whole point of Unicode is to support multiple languages.

Really, if you're not targeting an international audience, then why bother with Unicode? Save yourself some memory.

Quote:

Original post by raydog Quote: Original post by petewood Quote: Original post by raydog The text strings just won't be in Japanese. What will they be? Are you sure? English, of course. The whole point of Unicode is to support multiple languages. Really, if you're not targeting an international audience, then why bother with Unicode? Save yourself some memory.

Because some of us do like to use different languages than English, wether you like it or not? I don't like having to use Romaji when I could have used Hirigana or Kanji.

And wether your program supports it or not, I can type Japanese into any windows control you use. It will drop to question marks when I'm done typing. Do you have any idea how annoying that is, and how much I want to severely maul the programmer who wrote the program? What is particularly annoying is when the program asks for a file and I want to use one that has a Kanji filename.

I recently changed a large project to Unicode, and here is what i used to figure things out

Read this first, its a very good primer that answers most upfront questions.
http://www.flipcode.com/articles/article_advstrings01.shtml


Then read this, actually, refer to it when you need info on a specific string functions.
http://www.metagraphics.com/index.htm?page=pubs/mgct_language-portable-code.htm


Hope that helps.

Quote:

Original post by Erzengeldeslichtes And wether your program supports it or not, I can type Japanese into any windows control you use. It will drop to question marks when I'm done typing. Do you have any idea how annoying that is, and how much I want to severely maul the programmer who wrote the program? What is particularly annoying is when the program asks for a file and I want to use one that has a Kanji filename.

Yes, that's a very good point. Typing in Unicode characters in a common dialog box that opens
and saves files probably won't work as expected in a program that doesn't support Unicode.

Hey, if you're writing a program like Explorer that needs to display a Unicode system file directory,
then yeah, it might important, but I'm not. :)

Quote:

Original post by raydog Quote: Original post by Erzengeldeslichtes And wether your program supports it or not, I can type Japanese into any windows control you use. It will drop to question marks when I'm done typing. Do you have any idea how annoying that is, and how much I want to severely maul the programmer who wrote the program? What is particularly annoying is when the program asks for a file and I want to use one that has a Kanji filename. Yes, that's a very good point. Typing in Unicode characters in a common dialog box that opens and saves files probably won't work as expected in a program that doesn't support Unicode. Hey, if you're writing a program like Explorer that needs to display a Unicode system file directory, then yeah, it might important, but I'm not. :)

I'd rather not try to predict where the user will want to type Unicode text. Better to use it from the beginning than to hear complaints later on.

Quote:

Original post by raydog Quote: Original post by Erzengeldeslichtes And wether your program supports it or not, I can type Japanese into any windows control you use. It will drop to question marks when I'm done typing. Do you have any idea how annoying that is, and how much I want to severely maul the programmer who wrote the program? What is particularly annoying is when the program asks for a file and I want to use one that has a Kanji filename. Yes, that's a very good point. Typing in Unicode characters in a common dialog box that opens and saves files probably won't work as expected in a program that doesn't support Unicode. Hey, if you're writing a program like Explorer that needs to display a Unicode system file directory, then yeah, it might important, but I'm not. :)

Doesn't need to be like explorer.
For example: Winamp uses Windows' Open Dialog Box to get the MP3s. From here, I can get to any of my songs with title in kanji. This works perfectly fine and winamp can open it fine because windows is doing all the work, and windows supports unicode. However, once they're in the play list (since Winamp doesn't support unicode) I can't tell which song is which, they're all gobly gook. I wish to severely maul the programmer who wrote the text code they're using (according to the winamp guys, they can only display characters in the current code page because the text rendering engine they're liscensing only renders textures in the system code page). I've been forced to use iTunes as my primary player.
Another Example: AIM can't use unicode, and so I can only send/recieve messages in ascii. This is annoying. I do my best to get people to use MSN Messanger instead, but still have to use AIM for those who don't. (AOL: "It's America! Just because it's a melting pot of the world and has people speaking and writing every concievable language doesn't mean they want to communicate in anything but English!")

I'm in the United States. I doubt you'd consider me an "International" user. If you did, you'd be using unicode anyway. And as a "non-international" user, I'm asking you to use unicode. I'm not asking you to translate your English strings into 10 languages, I'm asking you to let me name things in 10 languages and have your program not care what language it's in. It's not that hard to do, and it doesn't take up that much memory. (Each string takes up twice as much space. If this increases your program size significantly, you have WAAAAAAYYYY too many strings. Compared to code, meshes, textures, sounds, movies, the doubling or even quadrupling of string size should be insignificant.)

[Edited by - Erzengeldeslichtes on March 15, 2005 2:21:49 AM]

It is really annoying that 128bit ascii stuff is still around, with unsafe function to manipulate things.
But unicode have made a mess out of something that should have been a nice standard I feel, because you'll have to read a ton of docs, and take care of alot of formats...

It is not about translating your program to different languages, but about having ppl with different language be able to type in their language.
...just think of notepad...

The downfall of c++, open source, linux and stuff could be as simple as lack of support, use and understanding of strings... 1byte ascii encoding must die. (even in unicode, utf-32 the values from 00 00 00 00 to 00 00 00 FF is ascii I belive, so it should be simple).

If I want to make my own string object, then the internal workings should be UTF-16 big endian, since this would have to deal with less conversion towards windows.

Todays question:
The use of L"...", TEXT"...", _T"...", what are they really doing? just pad the characters with 00 first?
So if a greek person would write

tchar foo_str[] = TEXT"Some greek letters I dont know of";

Would it come out right? I guess not since .h and .cpp files is just ascii .txt files really... So L, TEXT and so on is just for those who want to convert ascii to unicode? by padding?

I think your making your self think that its harder than it is.

To use Unicode on windows is very easy. Just define UNICODE and _UNICODE, there are the two definitions as one is Micorsoft's for the Platform SDK and the other is for the C++stdlib.

Then Instead of using char everywhere just use TCHAR( or wchar if you know you'll be strictly UNICODE ) for all strings that should be affected by teh encoding, almost all will, occasionally you'll have strings that should be 8 bit ascii.

Change all C++stdlib methods to use tchar methods, ie strcpy to _tcscpy.

Finally change all compile time string constants to use the _T() macro's. You don't need to worry about encoding if you do this. You'll use UTF 16 internally.

I'm afraid it was me who confused you by mentioning that if you want to port your program this approach will bring you a world of hurt. If its just a windows program this will do.

Cheers
Chris

in ascii you will terminate strings with \0

what is the termination code in unicode? or when using w_char?

another one...
int foo_function( w_char *str, size_t len )

if I use the function like this

foo_function( TEXT("Some text"), sizeof( ??? ));

how do I find the number of bytes sent?