🔒 Locked
FILE*
Advertisement
Original Post
When you open a file using fopen, is the file actually being streamed from disk or loaded in to memory all at once? If streamed then, every read and write required disk access. I'm trying to create a file i/o library and was thinking of loading the whole file into memory, allow user to manipulate it, and then save the file to disk if he or she wishes. But one problem with that is large files could very easily consume a large amount of memory. Any suggestion?
The CRT buffers some of the file as required. My version of the CRT (MSVC 2003) buffers 4k at once. So when I open the file and read 1 byte, it reads 4k, and just returns the first byte. So the next 1023 bytes are effectively free to read.
After you read those 1023 bytes (or you do an fseek()), the OS needs to read from the disk again.
If you're only targetting Windows, you might want to look into memory mapped files. They allow you to just get a pointer to the first BYTE of the file, and the OS automagically pages parts of the file in and out of memory as needed.
There's similar functions for linux, but it means you have two codepaths.
After you read those 1023 bytes (or you do an fseek()), the OS needs to read from the disk again.
If you're only targetting Windows, you might want to look into memory mapped files. They allow you to just get a pointer to the first BYTE of the file, and the OS automagically pages parts of the file in and out of memory as needed.
There's similar functions for linux, but it means you have two codepaths.
It's technically implementation defined. However most implementations do buffered reads. For example, on the first read, it'll actually read say a kilobyte no matter how small the request is, and then subsequent reads will receive data from the buffer until the buffer is empty.
If you want to manipulate an entire file at once via memory, it's more reasonable to use memory mapped files. However, memory mapped files require platform specific functions.
If you want to manipulate an entire file at once via memory, it's more reasonable to use memory mapped files. However, memory mapped files require platform specific functions.
Is there anyway to know the file size, without simply test reading every byte until the EOF and do byte count? Avoiding platform specify methods of course.
Some files don't have sizes, for they generate data as you read it. Finding the size of a file it platform-specific. You might get by, for ordinary files, by opening the file in binary mode, doing a seek() to the end of the file, and then a tell() to get the position. It is not guaranteed to indicate the file's size though (it is explicitely not when opening in text mode), but the odds are in your favour.
Topic Locked
This topic has been locked by a moderator. New replies are not allowed.
Sign in to reply to this topic.
Advertisement