Binary Files - C++ - Complex!

From http://www.cplusplus.com/reference/iostream/istream/getline.html:

Quote:

The number of characters read by this function can be obtained by calling to the member function gcount.

http://www.cplusplus.com/reference/iostream/istream/gcount.html

I'd suggest A) changing the format or B) changing how you parse the file.
Either way you should consider a "size of" marker before your binary fields so you know how much data is there, it will may your life a lot easier.

A) format change, put each field in the record on its own line. This just makes the parsing a lot simpler.

B) Parsing change. Instead of reading it by line, read it by token. Assuming tokens are just words like "file: " or "c:/my.tga" you can break it up based on delimiters like " " or "," or "<", whatever fits. Thus read the whole file token by token, and parse out meaning one at a time. That way you don't need to know anything except what the token means about the following data.

But, without more info on the file layout, it is hard to give specifics on what would be best.

KulSeram, unfortunately I can't change the format. I'm actually reading COBOL data files. That's why we have this 'mixed' layout. In fact, the layout isn't mixed, but one fixed layout for each file (not exactly because of redefinitions, but still). The problem is that my app is suppposed to be multi-purpose, that is, it shouldn't need to know beforehand the exact layout of the record.

Thanks, though.

Quote:

Original post by beun From http://www.cplusplus.com/reference/iostream/istream/getline.html: Quote: The number of characters read by this function can be obtained by calling to the member function gcount. http://www.cplusplus.com/reference/iostream/istream/gcount.html

beun, thanks a lot! I think this will solve this part of my problem.

Thanks again.

No problem [grin]

Quote:

Original post by HeavyStorm I'm trying to read a mixed (text|binary) file using STL ifstream.

This isn't provided for; specifying both ios::text and ios::binary for a stream doesn't make any sense. When in doubt, go with binary.

Quote:

Because I don't know X. Since we're talking about a text file (with binary fields on each line) I do, however, have an EOL to indicate when the record has ended. There's a small problem with the fact that the EOL might appear inside one of the binary fields, but let's ignore this at first.

This isn't really ignorable. You need to think about your data format. What are you really trying to store? Why are the binary fields binary? Why are the lines text?

Quote:

My line terminator would be 0x0A on most files (Unix style), but that can be configured in the application.

You have to decide what you want here.

If you want the line termination to be a specific character, then put your foot down. If you want it to be the natural line ending for the platform, you really need to use a text file, and encode your binary data in a text-friendly way. If you want the user to decide, then the file has to contain some information (metadata) at the beginning to say what the user decided, or else the user will have to tell the program every time you open the file, too.

Quote:

Where MAX_RECORD_SIZE is a constant that defines the maximum size for any given line. Of course, this is a very large buffer.

You don't need to do this; use a std::string to hold the line, and use the free function std::getline(), which reads into a std::string.

Quote:

HOWEVER, I must then findout the size of the line that I read. I thought that this would be accomplished simply by looking up the 0x0A on the file, but I found out that getline doesn't push the terminator on the read bytes. That's where lies my problem, how to determine how many bytes were read?

The input is null-terminated in this case, so you could look for the 0x00. However, that doesn't help you if your binary data contains 0x00. :)

Quote:

because on ANSI C I could do this very easily (and that's probably what I'm going to do if I can't find an answer).

I'm willing to bet that you couldn't really, because it's obvious that you have far more design issues than implementation issues here. What is actually in your file?

Quote:

Original post by HeavyStorm KulSeram, unfortunately I can't change the format. I'm actually reading COBOL data files. That's why we have this 'mixed' layout. In fact, the layout isn't mixed, but one fixed layout for each file (not exactly because of redefinitions, but still). The problem is that my app is suppposed to be multi-purpose, that is, it shouldn't need to know beforehand the exact layout of the record.

I think the best solution would be to provide the layout of the file as a separate parameter to your class some how. I've worked with COBOL data files as well (actually it was a network protocol based on the layout of COBOL data structures, but it's essentially the same thing), and it's not pretty... it's much easier to know the format of the structure before-hand.

My protocol was generic, too (in that it could work with any data structure) but the tool which generated the COBOL code was also used to spit out a "layout" file that my program was able to parse and interpret the data structures that way.

People, First of all, thanks a lot for the replies. I agreed with Zahlman that it seems that I have more design issues than programmatic ones (oh, and it's cool to know you still roam the forums, Zahlman!). As Codeka pointed out, COBOL file layout sucks. Here's how things goes: my application is being designed to work with such files, ones that have been downloaded directly from the mainframe and others that are generated through Micro Focus COBOL compiler on Unix systems.

The purpose of the application is a little bit crazy: what I'm writing is a diff, mostly like a Unix diff, to work with these files. We have very large data files (> 1G) and we need to compare results between two different executions. The comparison must also point which record was different, so it isn't enough to simply make a binary comparison. Even further, when possible we use another app to apply a layout to each of these records so that we can identify which column was changed.

COBOL sequential files have a number of particularities; the first one is that mostly those files have fixed length records – something called lrecl. Another particularity is that some files have variable records, which have their length identified on the first two bytes of every record. That said, I also have some eol-terminated files that I must diff. Those files are database “unloads”, generated by yet another tool that have this sole purpose. So my diff tool must read 3 different kind of files, each one to be treated as text even though they contain binary data.

On the variable and fixed length type of files, there isn’t a problem. I simply know how many bytes I should read. In case I get a EOF before that, either my file is corrupted or the informed lrecl is wrong. Of course, there are some adaptations to the usual algorithm that I must make – COBOL strings aren’t null terminated – so I must force my reads to go through all lrecl bytes. Since I’m writing a diff, my hash function takes a length parameter, for example. And one can guess that the zeroes makes crazy stuff on a hash function (not really, though).

On the eol-terminated files, I have the problem stated above; I have a eol character that can appear anywhere on the record. That means that I can’t decide if it’s a value or a eol mark.

So far, with the help of the gcount method, my problems have been dealt with… The diff tool was working before with eol when there weren’t any binary fields within the records, but now with the next adaptations the tool will become more full-fledged.

And in case you are wondering why I don’t simply use the Unix diff to do that, well, layout questions aside, Unix diff will load a lot of stuff in the memory – it does so until it finds a match between two records. So, on worst cases, I may have two whole files loaded in the memory. That means 4G on some cases, 8G on another. 8G here is half the paging size of the server, enough to make all apps to go down. Even ssh froze once.

Well, again, thanks. I thought I should explain better what the hell I was doing before you all called me crazy. The good news is that, so far, it works.

Quote:

Original post by HeavyStorm Well, again, thanks. I thought I should explain better what the hell I was doing before you all called me crazy. The good news is that, so far, it works.

If you're not crazy now, I'm sure you will be soon [wink]