🔒 Locked

Unicode source code

Started by Lode Feb 1, 2008 at 7:21 AM 7 replies 1k views

Original Post

Feb 1, 2008 at 7:21 AM

It seems most programming languages use ascii source code. * String literals that would contain unicode characters in the source code seem to be disallosed usually. * In identifier names, they're of course also totally not allowed. * And there exists no programming language I know that would use symbols like ˚ (degrees), π (pi), Σ (summation), ∞ (infinity), ∩ (intersection of sets), and whatever else you can think of that is available in unicode to actually mean those things in the code. The only symbols used are the standard ASCII symbols like & * % $ # and such. I understand the technical difficulties: * Not all text editors properly support anything else than ascii. * Standard American keyboards also don't have anything else than those standard characters printed on them, and other language keyboards all have their own selection of symbols. Do you know the existance of programming languages that support unicode characters in the source code in any of the ways described above? Maybe C# or Java even allow it? I'm not sure. Would a language that would make real use of unicode characters, maybe even as identifiers, be able to have success?

choffstein

Feb 1, 2008 at 7:29 AM

APL has a rather strange character set...all of which is found in unicode. For this reason, the derivative languages J and K had to come up with ways of representing the same notions with different character representations.

ToohrVyk

Feb 1, 2008 at 7:36 AM

This has been the subject of an april fools joke.

SiCrane

Feb 1, 2008 at 7:41 AM

Here's an interesting tidbit: the Unicode standards themselves contain functionality for reasonable use in programming languages. The Unicode standards define various character classes to identify code points suitable for use as identifiers: ID_Start and ID_Continue; these were then updated to XID_Start and XID_Continue, though it's still possible to use ID_Start and ID_Continue. The standard also recommends the use of NFKC to normalize identifiers. This prevents things like pathfinding and pathﬁnding as showing up as different identifiers.

For more details see TR15 and TR31 in the current version of the Unicode standard.

Unfortunately, the only language I know of the takes advantage of these facts is Indigo, which is a research language that hasn't been released to the public.

edit: Though it is under consideration for Python 3000.

DeadXorAlive

Feb 1, 2008 at 7:45 AM

Fortress makes use of special symbols / mathematical notation. There must be more languages that support Unicode, I know that the D programming language supports several unicode encodings in the source, but it doesn't give any non-ascii characters special meaning.

benryves

Feb 1, 2008 at 8:15 AM

C# handles Unicode characters in identifier names and string literals fine (if you edit a source file and add a unicode character it'll prompt you to change the encoding to something more suitable).

The text editor can get a bit confused, though, if it has to substitute in characters from another font, and ends up using that font for other characters nearby.

Extrarius

Feb 1, 2008 at 8:41 AM

One huge difficulty with such an idea is that most fonts don't include many non-ASCII characters. I really wish the Unicode group would pay all the companies that donated the example images and make it into a font (in as many file formats as possible). Are there any fonts that include every Unicode character? If so, are any of them suitable for programming?

benryves

Feb 1, 2008 at 8:45 AM

Quote:
Original post by Extrarius
One huge difficulty with such an idea is that most fonts don't include many non-ASCII characters. I really wish the Unicode group would pay all the companies that donated the example images and make it into a font (in as many file formats as possible). Are there any fonts that include every Unicode character? If so, are any of them suitable for programming?

Most decent text editors or viewers should substitute in characters from similar fonts if the main font doesn't cover them, which is the reason for the odd text rendering in the VS screenshot I posted - it seems to have switched to using MS Gothic (a font with extensive Unicode coverage) for the arrow characters.

Extrarius

Feb 1, 2008 at 8:51 AM

Quote:
Original post by benryves
[...]Most decent text editors or viewers should substitute in characters from similar fonts if the main font doesn't cover them, which is the reason for the odd text rendering in the VS screenshot I posted - it seems to have switched to using MS Gothic (a font with extensive Unicode coverage) for the arrow characters.

Yeah, but that is obviously a suboptimal solution, especially for programming, since it kind of breaks the idea of having a fixed-width font if you're going to mix in other fonts that either aren't fixed width or otherwise mess with the appearance such that it isn't uniform anymore.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.