🔒 Locked

[java] Strings and unicode characters

Started by Last Attacker Apr 4, 2005 at 5:17 AM 5 replies 1.1k views

Original Post

Apr 4, 2005 at 5:17 AM

I've written a web-page compiler (not like a C++ compiler ~:| ) but what takes all the directories linked with a web-page (which you've saved from IE or whatever) and copies them all to a newly created directory. It then deletes those other directories. Then it loads all the data from the html files, replaces the images links and rewrites the html files. Everything works fine except for webpages using unicode characters. It makes weird stuff in the place of the character where it originally was in the html file, like the pound sign shows ║╧§ (something like that). Any thoaghts on this? Thanks.

Dean Harding

Apr 4, 2005 at 5:34 AM

What's your code for opening and reading the files? Most likely it's opening the file assuming it's in whatever codepage your computer is in (windows-1252 for English) and thus it'd screw up UTF-8 encoded files.

Theoretically, you should scan the file for the meta tag that describes the encoding (<meta http-equiv="Content-Type" content="text/html; encoding=utf-8" /> or something like this), but if they're all your own HTML files you could safely assume UTF-8 encoding or whatever you decide to standardize on.

Last Attacker

Author

Last Attacker

Apr 5, 2005 at 4:48 AM

I've done it plainly :

file_read = new BufferedReader(new FileReader(html_file));length = (int)html_file.length();data = new StringBuffer(length);while(file_read.ready()){   data.append(file_read.readLine());   data.append('\n');}

EDIT : html_file is the File object holding the FileName of the desired HTML file.

[Edited by - Last Attacker on April 5, 2005 5:48:00 AM]

Dean Harding

Apr 5, 2005 at 5:11 AM

Quote:
Original post by Last Attacker
file_read = new BufferedReader(new FileReader(html_file));

This is your problem. According to the Java SDK docs:

Quote:

public class FileReader extends InputStreamReader

Convenience class for reading character files. The constructors of this class assume that the default character encoding and the default byte-buffer size are appropriate. To specify these values yourself, construct an InputStreamReader on a FileInputStream.

So given this, you should be using the following:

file_read = new InputStreamReader(new FileInputStream(html_file), "UTF-8");

Last Attacker

Author

Last Attacker

Apr 5, 2005 at 5:21 AM

Ok, thanks.

So I start off by opening the html file via RandomAccess, look for the character-encoding-tag, then reopen the file via InputStreamReader and the encoding tag.

Thanks Dean Harding.

Dean Harding

Apr 5, 2005 at 5:28 AM

Yeah, that would be the most flexible way (it's the way the browser works out the encoding). I'm not a Java expert, but I'd wager that you should be able to pass whatever encoding the Content-Type value specifies as the second parameter to the InputStreamReader constructor and it should work (that is, Java should support all the major encodings that people'd use to encode HTML files).

Last Attacker

Author

Last Attacker

Apr 5, 2005 at 5:32 AM

Don't worry, its cool.

Thanks for your help, I appreciate it.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.