difference between binary file and text file

A text file is a binary file whose bytes can be read to fall within the limits of some character set (be it ASCII, Unicode, or even other multibyte charsets). The only difference is that imposed by the functions which read from and write to the file. When Linux is "wrong", it may be trying to interpret text files from ASCII or another non-Unicode charset; since Unicode has a different range than ASCII, it fails.

To detect whether a file is binary or text, then, requires knowing the character set the text would be in. You then read from the file and see if any of the bytes (or bytesets) falls outside of the specified character set.

[Edited by - TDragon on October 25, 2005 10:18:55 AM]

Okay, this is a big pet-peeve with me. "How to ?" is not a valid method of formulating a question. It may be an acceptable way of starting the title of a book*:


It is not acceptable for starting a question. I reallize English is not your first language, but I actually see this from native English speakers the most.




*it was actually rather difficult to find an image that would be appropriate for gdnet. A search for "how to" on amazon.com returned mostly books on sexual activity.

I thought it had to do with the CR/LF characters at the end of a line.

In binary mode, they'd read just as they are in the file.

In text mode, on windows, a CR/LF would come back as a new line. While on unix (linux?) it would only need either the CR or LF, so one of the CR/LF characters would be stripped from the input.

There is actually no difference in the file. Its just a mode on how the ANSI C functions operate on the file to handle those CR/LF characters.

Something to that effect.

That annoys the hell out of me too, midnight. Unlike you, I notice it most often in orientals and Europeans. They drive me insane. THEY ALL DO IT.

How to rephrase the question so that it's the same question but without "how to" at the start? :D

Quote:

Original post by JoeyBlow2 I thought it had to do with the CR/LF characters at the end of a line. In binary mode, they'd read just as they are in the file. In text mode, on windows, a CR/LF would come back as a new line. While on unix (linux?) it would only need either the CR or LF, so one of the CR/LF characters would be stripped from the input. There is actually no difference in the file. Its just a mode on how the ANSI C functions operate on the file to handle those CR/LF characters. Something to that effect.

I think the original question wasn't how are the operated on differently, but if you have a file of unknown type, how to determine if it is a text file, or binary file (or probably just more generally how to determine if a given file is some form of text file).

There really is no surefire way to determine that though. My best guess at it would be to look for patterns you'll frequently find within a text file, and choose several different patterns to look for.

For example, look for frequent occurances of the space character code (32 decimal in ASCII), with less frequent occurances of "periods,double spaces" as well as other frequently used words within text (though this makes it mostly language specific as well) such as

a 32,97,32
the 32,116,104,101,32
or 32,111,114,32

etc. The more patterns you search for, the more reliable the test will be. Look for ASCII and Unicode patterns (and any other character set you may expect) and you should be able to determine which character set is used as well. Run samples on various existing text files to generate histograms of occurances and use that data as a baseline for profiling unknown files.

Of course, that's an aweful lot of work to do it automatically. The alternative is to load the file into your favorite raw text editor and see if it's garbage or readable text, unless you need to be able to determine it programmatically for some reason.

Replace "to" with "do I" or "can I" or "should I". Alternatively, "What code should I use to...".

Isn't the unicode one fairly trivial... I forget which way round it is but there's basically big-endian and little-endian... both have a 2-byte header: one starts with 0xFE,0xFF and the other 0xFF,0xFE.

I'm not so sure on the EOL's, but I'd imagine that you could do similar pattern matching - looking for CRLF, LFCR, CR, LF etc...

Jack

Quote:

Original post by Lode How to rephrase the question so that it's the same question but without "how to" at the start? :D

As TDragon stated, you could rephrase the question this way: "How do I detect the difference between a binary and a text file?" However, the following might be better: "In my program, how can I detect if a given file is text or binary?"

How about reading the first 100 characters. If atleast, say, 90 of them are between 0x20 and 0x7F (or 0x0A and 0x0D (CR and LF respectively)) then call it text. Kind of crude, but in most cases, it should work.

How to drive Kevinator crazy in three easy steps.

Quote:

Original post by Lode How to rephrase the question so that it's the same question but without "how to" at the start? :D

The problem is that your questions don't state the object that is doing the verb. In the quoted case, the verb is 'rephrase' and the object that will rephrase is you. An appropriate way to rephrase it would be to instead say "How can I rephrase the question.." or "How could I.." or "How would I.."

I know other people already answered you, but they didn't state the problem so much as to just offer solutions. Knowing that you're missing nouns might help you better phrase things in the future =-) I'm guessing your native language allows you to omit certain nouns, and it probably does so because they're conjugated into the verb.

A text file is simply a special case of a binary file, containing only ASCII characters (or other character types). These days it's like some people think they're something else.

The only real way to know is to read it in and have a look. (In binary mode of course)