Skip to main content
gamedev.net
🔒 Locked

file loading

Started by Lode Oct 9, 2006 at 5:13 PM 6 replies 1.2k views
Original Post
Lode
Lode
Is the following code the shortest way to load a file into an std::vector with a C++ fstream? 

  void loadFile(const std::string& filename, std::vector<unsigned char>& buffer)
  {
    std::ifstream file(filename.c_str(), std::ios::in|std::ios::binary|std::ios::ate);
  
    //get filesize
    file.seekg(0, std::ios::end);
    int size = file.tellg();
    file.seekg(0, std::ios::beg);
    size -= file.tellg();
  
    //read contents of the file into the vector
    buffer.resize(size);
    file.read((char*)(&buffer[0]), size);
    file.close();
  }
I'm a bit worried that the get filesize stuff is pretty clumsy. Is it?
Homepage | LodePNG
Fruny
Fruny
No, it is not:

#include <fstream>#include <iterator>#include <vector>std::ifstream ifs("foo.dat", std::ios::binary);std::istream_iterator<unsigned char> begin(ifs);std::istream_iterator<unsigned char> end;std::vector<unsigned char> vec(begin, end);


Note that loading a large file of unknown size into a deque is generally faster than into a vector (no reallocation). Using a istreambuf_iterator may also offer a speedup.

The "filesize" stuff is indeed clumsy, since the only truly reliable ways of getting the size of a file are platform-specific. In binary mode, you should be safe enough, but you never know...
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." — Brian W. Kernighan
Evil Steve
Evil Steve
Quote:
Original post by Fruny
No, it is not:

#include <fstream>#include <iterator>#include <vector>std::ifstream ifs("foo.dat", std::ios::binary);std::istream_iterator<unsigned char> begin(ifs);std::istream_iterator<unsigned char> end;std::vector<unsigned char> vec(begin, end);


Note that loading a large file of unknown size into a deque is generally faster than into a vector (no reallocation). Using a istreambuf_iterator may also offer a speedup.

The "filesize" stuff is indeed clumsy, since the only truly reliable ways of getting the size of a file are platform-specific. In binary mode, you should be safe enough, but you never know...
Is there not some way of pre-reserving the vector then? I'd be tempted to use C file IO to read into a pre-resize()d std::vector (Yes, eww, I know), since it'd avoid the reallocation.
Fruny
Fruny
You have a few options:
* use seekg/tellg, or fseek/ftell and hope for the best.
* use boost::filesystem::file_size.
* load into a deque, get the size, then copy into a vector.
* load into a vector and do not worry about the resizing.

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." — Brian W. Kernighan
Zahlman
Zahlman
Quote:
Original post by Fruny
Note that loading a large file of unknown size into a deque is generally faster than into a vector (no reallocation).


Really? Isn't the deque constantly allocating "pages", and also suffering cache coherency issues? It seems to me like the real overhead with the vector is *copying* for reallocation, and that shouldn't be much in comparison (expected value ~2 copies per element).

Quote:
The "filesize" stuff is indeed clumsy, since the only truly reliable ways of getting the size of a file are platform-specific. In binary mode, you should be safe enough, but you never know...


Out of curiousity, what are the sorts of things a platform might do that could cause fseek()/ftell() magic to fail? And would it really have been that hard to build "a reliable way" into the standard library? (What exactly is boost::filesystem doing, anyway - just wrapping a bunch of known platform-specific tricks?)

Absolution
Absolution
Quote:
Original post by Zahlman
Out of curiousity, what are the sorts of things a platform might do that could cause fseek()/ftell() magic to fail? And would it really have been that hard to build "a reliable way" into the standard library? (What exactly is boost::filesystem doing, anyway - just wrapping a bunch of known platform-specific tricks?)


On win32, boost is using the GetFileAttributesEx function. You pass it a WIN32_FILE_ATTRIBUTE_DATA structure. This is probably the best way to query a file size on windows.

Edit: For POSIX it's using the stat structure and the stat function (all can be found in the MSDN.)
Aardvajk
Aardvajk
Quote:
Original post by Zahlman
Out of curiousity, what are the sorts of things a platform might do that could cause fseek()/ftell() magic to fail?


Not really a failure, but '\r\n' to '\n' conversions create a discrepancy between file size on disk and file size when read into memory for text files. The Windows API calls are as guilty of this as the fseek()/ftell() trick though.

Fruny
Fruny
Quote:
Original post by Zahlman
Really? Isn't the deque constantly allocating "pages", and also suffering cache coherency issues?


It has been my experience that loading in a deque is faster. That's all I can really say about it. With a large file, the vector will have the same cache issues as a deque would, as well as trouble if memory is fragmented.

Quote:
Original post by Zahlman
Out of curiousity, what are the sorts of things a platform might do that could cause fseek()/ftell() magic to fail?


Files larger than 4GB, and the fact that there is no guarantee (AFAIK, as usual) that the return type from tellg() be a type on which you can perform arithmetic, much less a long long. In binary mode, yes, you can expect the thing to represtent a byte offset, just like most vector implementations were using continuous blocks of memory even before it was required by the standard. But there is no absolute guarantee it will be so.
Platform-specific functions have the advantage that they know in which format the file is stored on the disk (if it even is). Knowing how much data it contains is part of the information it needs to even begin manipulating those files. Why isn't it so with C/C++ IO functions? Well, because not all files have a size, for starters. Standard input certainly doesn't. And yet, at least in C, it is a FILE*. In the UNIX world, almost everything is a file...

[Edited by - Fruny on October 10, 2006 3:57:00 PM]
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." — Brian W. Kernighan

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.