How do I save an std::string to a file if there are non-ascii characters present

No, that's not what the "Unicode" setting does. The "Unicode" setting in VS just #defines UNICODE project-wide. This can affect things like WinAPI calls, so that DrawText for example will be routed to DrawTextW (Unicode) rather than DrawTextA (ANSI).

std::string and std::wstring don't necessarily correspond to ANSI and Unicode strings, respectively. It's just that std::string uses a narrow characters (char) and std::wstring uses a wide char (wchar_t). On Windows, Unicode is represented using UTF-16, which is why wchar_t is 2 bytes. But since std::wstring isn't specifically UTF-16, size() and length() don't take into account surrogate pairs and the like.

The short of it is: Unicode in C++ is a mess. size() and length() will get you the number of code units in the string, not the number of code points. This is because neither std::string nor std::wstring know which encoding is being used.

I typically go with the old stand by of ofstream << string, but maybe that is just me.

Quote:

size() is the same as length()

Yes.

Quote:

gives you the number of characters in the string.

only if you use ascii. I am not sure but I think Unicode on windows usually means UTF-16. std::string is actually a template specialized on chars so length\size return number of chars and has nothing to do with characters. This also means that you could specialize it on something else to get size\length to represent something else. Mostly std::string is vector with a pretty face.

Quote:

The short of it is: Unicode in C++ is a mess. size() and length() will get you the number of code units in the string, not the number of code points. This is because neither std::string nor std::wstring know which encoding is being used.

Oh, so size() and length() are the length of the buffer? Awesome, that's what I was hoping (although I can see how it would suck if you actually wanted to iterate over each character).

Quote:

I typically go with the old stand by of ofstream << string, but maybe that is just me.

Does this work with binary streams? If so I will use it.

Thanks a lot both of you!

Quote:

Oh, so size() and length() are the length of the buffer? Awesome, that's what I was hoping (although I can see how it would suck if you actually wanted to iterate over each character).

size() and length() return the number of characters in the string, not the size of the buffer (of course in some circumstances the number of characters and the number of bytes in the buffer will be the same).

Quote:

Original post by jyk Quote: Oh, so size() and length() are the length of the buffer? Awesome, that's what I was hoping (although I can see how it would suck if you actually wanted to iterate over each character). size() and length() return the number of characters in the string, not the size of the buffer (of course in some circumstances the number of characters and the number of bytes in the buffer will be the same).

Number of code units, not characters. For example, "é" is one "character", but two "code units" (U+0065 U+0301). It can also be written as "́é" which is one code unit (U+00E9). Welcome to the wonderful world of Unicode :-)

Alright cool, and each code unit in UTF-8 is a byte so I can be like:

stream.write(str.data(), str.size());

and it'll work right. Thanks everyone, for your help.


Edit: and by work right I mean it will copy the entire buffer through the stream without clipping it off if there are multi-byte characters.

Quote:

Original post by Drakkcon Alright cool, and each code unit in UTF-8 is a byte so I can be like: stream.write(str.data(), str.size()); and it'll work right. Thanks everyone, for your help. Edit: and by work right I mean it will copy the entire buffer through the stream without clipping it off if there are multi-byte characters.

Correct.