Localization

Quote:

Original post by codemonkey47 I was wondering, how do you localize strings in a program so the user can select a language other than English? Are the strings just stored in a text file? Thanks for the help.

Obviously the strings will need to be stored somewhere. Storing them in a file will make localization easier, because you won't need to recompile in order to add a language or to fix a few spelling mistakes. Of course, you'll need to write a system that reads these string-files into memory, and you'll have to request strings from that system, instead of hard-coding them into the program.

The short answer: Yes.

The longer answer: Each localizable string is stored in a text file along with a short token that the game code uses to represent that string. For example:

PRESS_START="Press Start"

The game code would load the text file and separate it into tokens and strings, storing in a manner that allows quick retrieval. We use a hash table with the token as the hash key. Now that the file is loaded and stored in a data structure, use a function to retrieve the localized string, e.g.

const char *GetLocalizedString(const char *token);

To make it easy, we shorten this with a macro

#define LOC(t) GetLocalizedString(t)

Now use this whenever you want a localized string.

To use a different language, replace the string in the file, but keep the token the same. For example, here is a file in the language of Gibberish:

PRESS_START="Googoo startumup"

Now load Gibberish.txt instead of English.txt and the text shown to user changes to the selected language.

I would also recommend never storing the pointer to the localized string after retrieving it via the LOC() macro. Always use LOC() to retrieve the string every time you want to display the string. This allows you to easy change languages on the fly for simple testing.

As others have said, you just take every string in your program and localize it into all the languages you want to support. You then give it an ID, and use the ID in your code by passing into the localization system. It will then return the correct string depending on the currently set language.

However something that often comes up with localization is dealing with format strings, and you'll need to think about how you want to handle them. If you want to tell the user "You are carrying %d %s %s.", which could be replaced with "5", "red", and "apples", respectively, you need to way to properly localize that into all languages. For instance, in Spanish the adjective comes after the noun, where in English it's the other way around. So your system needs to be able to deal with that without requiring your code to special-case different languages (or at least transparently to the user).

Quote:

Original post by Sly const char *GetLocalizedString(const char *token);

Good start, but it trips over itself. Use wchar_t (which is 16-bit) at a minimum, or tools that can handle the full Unicode spec, such as the Unicode Consortium's own ICU library.

Quote:

Original post by Zipster For instance, in Spanish the adjective comes after the noun, where in English it's the other way around. So your system needs to be able to deal with that without requiring your code to special-case different languages

It is still much more than just swapping out strings, and even re-formatting them for the grammar in question.

You absolutely must consider general text layout, box sizes, and positions. Most game publishers and all the consoles will block a game that doesn't display the text properly in all available languages.

For other languages you will need to consider text flow direction, left/right alignment, and countless other issues.

What has been mentioned is a good start, but ultimately the product needs to be evaluated by people who actually speak and use the language, and it needs to look as though it were written for that language natively --- because it actually is.

Quote:

Original post by frob It is still much more than just swapping out strings, and even re-formatting them for the grammar in question. You absolutely must consider general text layout, box sizes, and positions. Most game publishers and all the consoles will block a game that doesn't display the text properly in all available languages. For other languages you will need to consider text flow direction, left/right alignment, and countless other issues. What has been mentioned is a good start, but ultimately the product needs to be evaluated by people who actually speak and use the language, and it needs to look as though it were written for that language natively --- because it actually is.

If you're localizing for dozens of languages then yes, there a lot more things to consider than just grammar and positioning. But for most publishers and the consoles EFIGS (English-French-Italian-German-Spanish) is generally all that's required. Beyond that you're trying to make your game accessible to a wider (namely the Asian) market. Always a good thing, but a lot more work as well.

I've thought about this before. Just store the English sentences in a file with IDs. At compile time have a program run which opens an HTTP request with google's translator. You simply enter the languages you want to support and have it throw back the translated versions into the file. Magically your program will support tons of languages and you can add strings in as you edit things without manually translating the sentences each time.

Quote:

Original post by Zipster If you're localizing for dozens of languages then yes, there a lot more things to consider than just grammar and positioning. But for most publishers and the consoles EFIGS is generally all that's required. Beyond that you're trying to make your game accessible to a wider (namely the Asian) market.

All three console vendors (Nintendo, Sony, and Microsoft) will fail your submission if text overflows your text box, they don't care if it is just EFIGS or if you have a longer list of languages.

Nintendo is (in my experience) the strictest of the bunch, they'll get upset about it --- but in a good way, naturally, since they are pushing for quality. When it comes to EFIGS, I hate German the most. Combine the language's natural concatenation and the user-entered test string "WWWWWWWWW", and Nintendo will always come back with a lotcheck violation. Even when you do it properly, they'll complain the words get too close to the margins, or they leave too much space in the wrapped lines, even when they know it cannot be split any other way.

And it isn't just Asia. My last title had 14 languages and all were European. The UI artists and loc team issued threats against the people marketers who demanded a Finnish translation.

So no, you must consider more than just swapping and formatting the strings, and think about more than "just Asia".

And on the matter of const char*, don't do it. Ever. It doesn't matter if you want EFIGS specifically, or just European or Western languages generally, writing code for 8-bit extended ascii is just not appropriate for so many reasons. When looking up strings for display to the UI, don't do yourself and other people the disservice of returning 8-bit strings. Even if you have just EFIGS, and even if you are on a DS or a cell phone, the localized strings for display should absolutely be (at least) wchar_t strings.

Quote:

Original post by frob All three console vendors (Nintendo, Sony, and Microsoft) will fail your submission if text overflows your text box, they don't care if it is just EFIGS or if you have a longer list of languages.

Obviously they're going to test all the languages your title supports and if they don't all display properly you'll fail the submission. That goes without saying.

Quote:

So no, you must consider more than just swapping and formatting the strings, and think about more than "just Asia".

For general localization there are a myriad of issues to contend with, I never said otherwise. But that's why I specifically chose EFIGS as an example, because they're the easiest languages to localize for (relatively speaking), since spacing and formatting are your biggest issues, neither of which are difficult to deal with from a UI standpoint so long as you keep them in mind during design. The more languages you support the harder it gets, that I don't disagree with.

Quote:

And on the matter of const char*, don't do it. Ever. It doesn't matter if you want EFIGS specifically, or just European or Western languages generally, writing code for 8-bit extended ascii is just not appropriate for so many reasons. When looking up strings for display to the UI, don't do yourself and other people the disservice of returning 8-bit strings. Even if you have just EFIGS, and even if you are on a DS or a cell phone, the localized strings for display should absolutely be (at least) wchar_t strings.

I agree, we use wide characters for localized text. I never said one should use otherwise [smile]

UTF-8 is a valid format to use for localized strings, just like UTF-16 is also a valid format. My personal preference is for UTF-16, but I am one voice in a large group of co-workers who can only think that UTF-8 is easier to support. In EFIGS, UTF-8 will be slightly smaller in memory use than UTF-16 (most characters will occupy one byte), but Asian languages will occupy up to 50% more in UTF-8 form (most characters will occupy three bytes). I fought the battle to use UTF-16 for a long time and just had to give in. Despite personal preferences, UTF-8 is still a valid format to use.

My previous project supported 19 languages including Chinese, Japanese and Korean. It used UTF-16 because that's what Microsoft Game Studios' localization tools use. All of our other projects have used UTF-8. Microsoft have been the toughest I have dealt with when it comes to localization, and that is because as a company Microsoft put a huge emphasis on creating software for a global market. Nintendo and Sony have a lot of work to do in this area. They don't even support locales yet.

Generally, UI elements should allow 50% extra for languages such as German. When supporting Asian languages, you get into a whole world of word- and character-break rules.

Quote:

Original post by frob And it isn't just Asia. My last title had 14 languages and all were European. The UI artists and loc team issued threats against the people marketers who demanded a Finnish translation.

You just made me remind when once I almost get into a flamewar with somebody from Spain because I had used a neutral dialect instead of Spain's (the game was in Spanish). Which is pretty much why I want the game my team is making to have several dialects for some languages because some players started becoming that demanding. Yes, it goes so far in some places (I've also heard complains about games getting subbed but not dubbed).

I also remember somebody mentioned in a Gamasutra article, that complained that voices in a game weren't phonetically perfect (nobody spells all words perfectly, come on) and that he wasted all his effort of teaching his daughter to pronounce properly thanks to the game (if you communicate enough time with your children that shouldn't happen >_>).

Here is a more in-depth gamasutra article on localization of text. I wrote a pretty shitty implementation of the system they use a couple of years ago, and it wasn't more than a couple hundred loc. You should be able to whip one up pretty easily.

Quote:

Original post by visage Here is a more in-depth gamasutra article on localization of text. I wrote a pretty shitty implementation of the system they use a couple of years ago, and it wasn't more than a couple hundred loc. You should be able to whip one up pretty easily.

The meta-grammar was pretty much what I was getting at earlier, however I didn't know about that (excellent) article and it would have taken forever to write up a post going into much depth. MMOs are one of the more extreme cases as well, since there are thousands of objects and hundreds of way that they can interact with each other, the world, the players, etc. Most games usually won't need something that fancy, however it seems like a really cool system to have in your bag of tricks.

Quote:

Original post by frob And it isn't just Asia. My last title had 14 languages and all were European. The UI artists and loc team issued threats against the people marketers who demanded a Finnish translation.

I'm a bit surprised that the Finnish language (my native language) could specifically be a problem.

It is written left to right.

And the only characters which we use in the Finnish language and which are not in the English alphabet are ä and ö (a and o with umlauts, I guess this forum software was not written well to handle these characters which are in the extended ASCII character set... at least they don't show correctly on my screen edit they didn't show up correctly in the preview but in the thread they show up correctly).

We also have what we call the "Swedish o" å (a with a small o on top of it) in our alphabet but if you write Finnish then that is not used. Swedish is an official language here and I guess that is the only reason why å is in our alphabet.

I don't see how this language could be problematic. I guess it had more to do with people just opposing doing more work - or so they assumed it would be more work, if they do German, then Finnish is surely not a problem from a technical point of view.

You of course must have people who speak the language to translate to it and that surely is not a problem with Finnish. Maybe the problem was that your people didn't want to do the extra work to find the Finnish people to do the translation, as I can't believe the reason was technical. It's just the two characters with the umlauts which are not in the English alphabet, and if you do German (or Swedish) then you already have them supported (the Finnish language is not a Germanic language like the other two mentioned, though, don't let the fact that there are these two common characters fool you into thinking it is).

[Edited by - reptor on February 17, 2009 7:15:20 PM]

Quote:

Original post by reptor Quote: Original post by frobAnd it isn't just Asia. My last title had 14 languages and all were European. The UI artists and loc team issued threats against the people marketers who demanded a Finnish translation. I'm a bit surprised that the Finnish language (my native language) could specifically be a problem.

You are correct, there is nothing technically difficult about adding it.

The annoyances were the number of agglutinative words, the script didn't translate well into the language, the overlap that we were already supporting Swedish, English, and German, and that we were not going to be sold or marketed specifically in Finland.

I don't understand how agglunative words are a problem - you are going to have native Finnish speakers to do the translation and it will not be a problem for them.

It is a problem if someone who doesn't actually speak Finnish tries to translate to it :) I hope that's not what your team was trying to do. It's like using Google Translate... it doesn't understand context or sentences so it just translates word-by-word and the result is usually very bad. It doesn't get the agglunative words right. But for a native speaker of the language, who also speaks English, it is not a problem to translate from English to Finnish, agglunative words or not. So I don't really get this "agglunative" excuse. In fact I think it is an invalid excuse which can be ignored :)

Maybe there is something I would need to try to do myself before seeing what the problem is. But at the moment it sounds like the problem was not the language but the number of languages the team was told should work in the product.

One thing which I want to ensure is understood is that the Finnish and Swedish languages are not close to each other - you should certainly not assume that someone speaking Finnish would understand Swedish. Or vice versa. Swedish is an official language here in Finland but it is only so because this land was once part of the Swedish Kingdom, the languages have a few common words, yes, but they are very far from each other. My estimation is that the clear majority of the people whose native language is Finnish are unable to have discussions (longer than a few simple words) in Swedish. English and Swedish are much much closer to each other than Finnish and Swedish (I guess thanks to the Vikings). If you want to think that you can cover more countries by just supporting Swedish, then the countries would be: Norway, Sweden and Denmark, maybe also Iceland, but *not* the Finnish-speaking people in Finland (there is the Swedish-speaking minority here which would benefit, but it is a small group of people compared to the whole population). Those are the Viking countries, they spoke the same language once upon a time but then their languages started to go down their own paths and so today they are their own languages, but they do share quite a lot. But do not make the mistake to include Finnish into that group. The Finnish and Estonian languages are close to each other, if you support the Finnish language then my estimation is that most of the Estonians will understand it. I can understand quite a lot of Estonian without ever having studied it, but this goes much better to the other direction, the Estonians understand Finnish much better than the Finnish understand Estonian. Sorry for this a bit off-topic paragraph but I don't think it's too far off :)

Quote:

Original post by reptor I don't understand how agglunative words are a problem - you are going to have native Finnish speakers to do the translation and it will not be a problem for them.

Agglutinative words are long. This causes a problem for word-wrapping in text boxes.

With our current project, the publisher has listed "Nordic" as a target language. My response to them was, "Which Nordic language?" I haven't got a response yet.

Agglutinative (never heard of that word before, but it seems appropriate) words are a fairly large problem for some European languages.

Quote:

Original post by Zahlman Quote: Original post by reptor I don't understand how agglunative words are a problem - you are going to have native Finnish speakers to do the translation and it will not be a problem for them. Agglutinative words are long. This causes a problem for word-wrapping in text boxes.

Especially on handhelds like the DS or PSP.

One particular block of text worked out to be a sentence of six agglutinative words, around 40 letters each. With word wrapping we had one word per line.

The text box looked bad since it was half empty, and Nintendo flagged it as a bug.

Hello! This is certainly a Babel kind of problem. I translated all the text of a game to spanish and I'm sure the original writer wouldn't have been pleased if he actually anderstood a bit of spanish. There are things that just simply cannot be translated literaly, there are many statements that depends upon the idiosincracy of the reader.