🔒 Locked

std::ifstream optimisations

Started by Dirk Gregorius May 17, 2004 at 8:17 AM 2 replies 600+ views

Original Post

May 17, 2004 at 8:17 AM

I read a huge ascii file ( > 2MB ) using the >> operator for formated file reading. The file is simply a rectangular matrix of values, eg. 1000 x 1000 floats. At the moment I do this:

std::vector<float> data;
data.resize( NUM_ELEMENTS );

std::ifstream is( FILENAME );

for ( int i = 0; i < NUM_ELEMENTS; ++i )
{
    is >> data[i];
}

In the debug build this is incredible slow. In the release build it take about two seconds to load a file of about 3MB. Are there any ways to optimize the loading code? BTW: Does a stream load the entire file into memory when opening it and do I load from the cached data? Best regards -Dirk

OrthoDiablo

May 17, 2004 at 8:39 AM

I am sure that ifstream does some kind of caching on the file. But in your case you may want to move away from formated reading and use binary files. This way you can read the entire file with one function call, which will probably be faster.

fredizzimo

May 17, 2004 at 9:17 AM

Are you using visual c++ 6.0? The fstreams are increadible slow in that. Get a better version of stl, like stlport, and you will see great speed improvements, without changing the code at all

Edit there's actually a fix for that(and many more fixes). But I still recommend stlport.

quote:

Fix to
(Added 13 October 1999.) The header defines template class basic_filebuf (among other things. Its member function _Init contains a bug that causes files opened by name to disable buffering unnecessarily, resulting in a serious degradation of performance. Change the test:

if (_Fp != 0 && !_Closef && sizeof (_E) == 1)
to:

if (_Fp != 0 && sizeof (_E) == 1)
to eliminate this problem.

[edited by - fredizzimo on May 17, 2004 12:01:29 PM]

Jingo

May 17, 2004 at 9:26 AM

You probably want to be using the stream buffer directly to increase speed. Does it need to be in a vector? You could do something like this if you can use a string..


std::stringstream str;
str << myinputstream.rdbuf();
std::string myfile = str.str();

If it needs to be a vector then you could use stream buffer iterators.


std::vector<char> k((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());//note the extra () around the first argument, they are not optional

Using stream buffers directly avoids the creation of sentry objects.

[edited by - Jingo on May 17, 2004 10:27:16 AM]

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.