Efficient disk I/O in Windows

I'd say these days sequential access is much less important for performance, at least on servers where you can control the hardware. That's because as long as your data is on an SSD the performance penalty for non-sequential access is relatively small - for example the best single drives will do 200MB/s of random 4k reads, and more more specialized hardware is even quicker.

If you really want to bypass the filesystem, I don't think it'd be a good idea unless you take over whole drives. The defragmenter is likely to mess up any direct access, if nothing else does.

Quote:

Original post by Adam_42 for example the best single drives

Apologies for going somewhat off-topic, but having owned one of those (the non-X2, not the very newest generation), I object to the word "best". On a 120GB drive which held 10GB total of infrequently written data, I was seeing occasional "bad block" messages in the event log after 3-4 months. Which, frankly, isn't acceptable, and for which there is really no excuse.
I did not actually notice any corrupted data or lost files, but seeing some more or less obscure, frightening failure message without really knowing what broke leaves an even more uneasy feeling.

The single basic requirement that a storage device (of whatever kind) must fulfill is that it reliably stores data for a reasonable time under reasonable usage. I would not object if the drive was 5-8 years old or if it was written to 50-60GB per day. Hey, sooner or later, things must break.
However, if you planned in a 90% "security marge", then it simply has to be 100% good, and there is no valid excuse nor any way to make up for that after the fact.

And with that, back to topic :-)

Regarding that Wikipedia quote, this is probably a candidate for "citation needed". While it may be possible that some systems do that, none of the "normal" ones that a million of people use every day does (such as for example one well-known Swedish SQL database, or one well-known Italian no-SQL key-data store, or one well-known Japanese no-SQL key-data store).
Writing blocks to disk directly has few advantages if any, but makes management a hell lot more painful, so I would not know why one would want to do that. Files are a good thing.

More often than not, sequential raw block acces is indeed of advantage, but as a matter of fact, for the particular case you just don't know. Raw "disk" may be a single contiguous real disk or partition, or a file image mounted as disk, a contigent of 6 disks with different sizes and speeds, or whatever. You might be writing into a snapshot while a backup is running at the same time (this is a very common thing), and you wouldn't know.

On the other hand, reading and writing raw blocks is a bitch to maintain compared to files. With files, if you want to make a backup, you take a snapshot, zip your database file and copy it somewhere, discard the snapshot, end of story. If you are using raw disk access, you must zip the whole partition, since you don't know which sectors might be used!
And in case your harddisk blows up or you want to upgrade hardware in three years from now, you must create a partition with the same size when restoring, copy the whole partition, and then somehow tell the database program to "enlargen" the data store.
Or, you must export and re-import your whole data. Good luck if your data is half a terabyte.

With files on a filesystem, you simply unzip what you have. Who gives a crap how big the partition is, it will just work.

And having said that, reserving large contiguous areas on the drive so your data is mostly or entirely sequential on-disk is something that every half decent filesystem (including NTFS) supports, and the actual performance overhead when using a high-performance FS (maybe not precisely NTFS) is neglegible.

Maybe the Wiki wording is just a bit misleading insofar as some database systems (but not all) will avoid going through the filesystem cache, for various reasons. Maybe that is what they meant.

Quote:

Original post by JonW The information that I'm most interested in is how a modern DBMS, such as SQL Server or Oracle, performs its I/O on the Windows platform nowadays.

Very carefully and slowly. Each time any piece of data is updated in the database, everything stops until OS confirms that the transaction has been fully committed to disk. So if you update one byte, the database will stop everything and wait for disk to confirm that it has been written correctly.

Nothing else matters, correctness and reliability take first place.

Quote:

so that the blocks can be sequentially read with minimal seek / rotational delays

That depends on data, not disk. If data and queries performed on it exhibit locality, then disk accesses will be local. If you do update *, then they will be all over the disk.

Quote:

in parallel with the CPU processing of the blocks being read.

Once you hit the disk you're slow already, and parallel can only be done to the degree transactions allow. For web, memcached is used. For large systems (banks and such), big iron is brought in. That one goes beyond 100k transactions per second if the price is right.


How to scale RDBMS? For heavy transactional data, buy fastest disks possible, put them in suitable RAID with lots and lots of RAM. For mostly read-only data, memcached. Horizontal partitioning may help as well.
Also, do not write your own RDBMS. Existing RDBMS databases have 40 years of top expertise in the field involved in them. Evaluate existing options - there is nothing you can do better.

Quote:

I'm interested however in retrieving large amounts of data using a custom format that would not fit well with a relational DBMS

Why not? What does the data look like? Format is completely irrelevant - what are access patterns? The rates? The latency? Transactions? Queriability? Relations?

Quote:

Apologies for going somewhat off-topic, but having owned one of those (the non-X2, not the very newest generation), I object to the word "best". On a 120GB drive which held 10GB total of infrequently written data, I was seeing occasional "bad block" messages in the event log after 3-4 months. Which, frankly, isn't acceptable, and for which there is really no excuse.

Which is a moot point. Hardware fails. Whether it fails once a month, once a year or once a decade - it fails. If data is even remotely important, this is addressed in the design via replication, mirroring and backups. Rate of failure is merely a cost factor to consider. But your disk *will* fail, that is as inevitable as death and taxes.

Quote:

Original post by samoth Apologies for going somewhat off-topic, but having owned one of those (the non-X2, not the very newest generation), I object to the word "best". On a 120GB drive which held 10GB total of infrequently written data, I was seeing occasional "bad block" messages in the event log after 3-4 months. Which, frankly, isn't acceptable, and for which there is really no excuse.

you relying on a single point of failure would be no excuse. but my drives (x1 and x2 and non intels) none behave like you described => your drive might be buggy (or needs a firmware update, there where some for x1 and x2). it might just be a missunderstanding of the drive and the os => nothing at all.

but the most important point you said: no data loss ever. just shows those drives work, and work great. i would never move back to platters anytime.


and to the op, i would try to get it fit into databases. consider the day that you suddenly need more than one server for scaling. with a database, that's (reasonably) simple. with your own file format, you suddenly have to code all kind of synchronization and all by hand.

Quote:

Original post by JonW that would not fit well with a relational DBMS

Would one of the non-relational DB's match your requirements, maybe?

Thanks for the opinions everyone.

Quote:

Original post by samoth Maybe the Wiki wording is just a bit misleading insofar as some database systems (but not all) will avoid going through the filesystem cache, for various reasons. Maybe that is what they meant.

I'm considering going the route of asking Windows not to cache my I/Os for me (unbuffered mode), because it is fairly simple to do your own caching (it only adds a few constraints on the sizes of buffers as well as their placement in memory and in the file). This Microsoft Research paper advocates turning off buffering when spreading the I/O operations across several drives -- unbuffered I/O was over 8x faster (there's some cool stuff in Microsoft Research by the way). Buffering doesn't make much of an I/O speed difference for a single drive, but CPU cycles per byte transferred drop to 1/10th of the amount with buffered I/O (the study used the .NET Framework).

You all have confirmed my suspicion that custom low-level I/O is way more complicated than what I need.

The reason a typical database doesn't suit my needs is because I need to look up 3D data with continuous levels of detail. Most RDBMS's today have quad tree support for spatial data of several dimensions, but the LOD aspect of my problem isn't well suited to records, and the data will be read-only in production. Also the data objects are really not that big, I just want to read them quickly :-)

I will have to go through those systems in your link phresnel; I don't know that any of them will do what I want here, but it looks like good information!

I think I have a direction now, but I still appreciate any comments on the matter.

Quote:

Original post by JonW because it is fairly simple to do your own caching (it only adds a few constraints on the sizes of buffers as well as their placement in memory and in the file).

Well...

"There are only two hard problems in Computer Science:
cache invalidation and naming things."

-- Phil Karlton

Quote:

Also the data objects are really not that big, I just want to read them quickly

How big is all of the data?

Quote:

Original post by Antheus "There are only two hard problems in Computer Science: cache invalidation and naming things." -- Phil Karlton

Yes, that is true, the second problem especially. In my case, I won't be doing much "caching", as I will be using an index to jump to a position in the 3D file and then do a sequential read of large blocks (64KB+) to extract some information from it. Once a block is processed in the sequence, it can be thrown out.

As for the size of the entire dataset--maybe a few hundred MB to a couple GB. That's not that big anymore, right? ;)

Another useful technique for fast writing if anyone is curious, is to preallocate the file. A good explanation of the technique from http://www.pcreview.co.uk/forums/thread-1484343-2.php):

Quote:

> NTFS zeros the file whenever you extend it. If you just use WriteFile() > to extend the file, the same zero'ing occurs. This is for C2 security > requirements, to prevent folks from extending their files over other > peoples files and taking control of the data. > > If you extend a file a little at a time, then for every extension: > Seek to cluster location > Zero out enough space for the addition > Seek back to the start of the write > Write the file data > > (so, 2 seeks for the write) > > Also, if something else has written to the adjoining cluster in the > interim since the last write, then the file will fragment. This happens > all the time in logging applications. > > So, if you pre-allocate the file, you only pay for the zeroing 1 time. > This will prevent any other files from intruding on your nicely sequential > allocation and optimize write speeds. If you only pay the price once, > then the the runtime performance is the same as bypassing the zeroing. > The clusters do not get zero'd until you call SetEndOfFile(). So the > order is: > > CreateFile > SetFilePointer (size of file you intend to create) > SetEndOfFile (commit the file) > SetFilePointer(beginning) > start writing > > No more zeroing will occur unless you overrun the file size allocated. > Also, fragmentation will be minimized. The zeroing happens very, very > fast b/c there is no seeking. A single seek to the start of the file, > then sequential writes so the writes occur at the disk speed.

You can use the SetLength() function in .NET for the same purpose.

Quote:

Original post by JonW As for the size of the entire dataset--maybe a few hundred MB to a couple GB. That's not that big anymore, right? ;)

So... why are we even discussing this? Why not just load everything into RAM?

Quote:

Another useful technique for fast writing if anyone is curious, is to preallocate the file.

Working with single file instead of many individual ones is always a good idea. Under NTFS especially it improves reads, possibly drastically. Might have something to do with handles or similar, I never investigated.

However - by using a single file you need to reinvent the file table and other file system details. It works well for reading - but as soon as trying to write all kinds of issues pop up, such as fragmentation.


IMHO, you might be approaching this from the wrong end. Performance is a matter of hitting various targets. And hardware scales quite far if needed.
1) Data is read only
2) Processing is IO bound
3) Processing is CPU bound
4) Data needs to be written

If 1) and 2), then compress data on disk, store everything into single image, keep offsets into file in memory. Don't optimize here too much, faster disk will help more than any time wasted on optimization.
If 1) and 3), then disk is not an issue at all. Put everything into single file, perhaps using uncompressed zip.
If 2) only, faster disk, SSD, RAID
If 3) only, consider using overlapped IO - it allows asynchronous file access and plays well with threads, making it simpler to work from multiple threads
If 3) and 4), consider fork/join, map/reduce model. Let the application just write into file. Post-process and collate these results elsewhere
...

And so on... Just trying to force some specific optimization may or may not work, it all depends on context.

And many of these problems can be solved with $50.

Quote:

I'm considering going the route of asking Windows not to cache my I/Os for me [...] This Microsoft Research paper

In my opinion, this paper and the related MS sources are somewhat unlucky, as they give people a wrong idea. There are some valid cases where unbuffered reads may be advantageous (such as in the paper that Jan Wassenberg posted here a year or two ago), but for the vast majority of people, turning off buffering is just a plain stupid thing. Well, in my opinion, anyway. Others might think differently :-)

It is quite correct that unbuffered IO can be faster than buffered, and it has less issues with the synchronous cache manager under Windows XP and 2000, also it uses DMA and uses no CPU, and it doesn't block. Ok, fine. So much for the propaganda.

Reality has it that reading from a conventional hard disk with totally cold caches (which in my case allegedly should give an average sustained read rate of over 100 MB/s) caps at about 50 MB/s buffered and unbuffered delivers 70-85 MB/s at best. Running a hard disk benchmark program reports numbers close to what's in the datasheet, but there is no way reading a real file gets anywhere close. It gets worse if you read several smaller files.
The solid state disk is about 3 times as fast on my system (should be 6 times according to the datasheet), but we still kind of play in the same ballpark.
Now, if all your prospective customers had systems running RAID with 16 stripes... but how likely is this going to happen.

So, unbuffered is faster? Yes it is if caches are cold, but usually, a considerable part of your data is in the buffer cache or "can be made to be" with proper hinting or explicit prefetching early enough.

Reading from the buffer cache delivers slighly above 3 GB/s on my system. This is still surprisingly slow as standard memcpy runs at more than three times as fast in the worst case, so you have to wonder what's going on in the buffer manager there, but never mind that... :-)
Now, 80 MB/s versus 3 GB/s, the choice isn't hard. Let's say a solid state disk has (depending on how much money you are willing to yank out) something between 200 and 600 MB/s, the choice still isn't hard.

Plus, unbuffered IO is a lot more complicated (have to be careful about alignment, etc) and error prone.

Add to this that 32bit processes (yes, they will die out one day, but that day is not today), doing their own caching can quickly take away a lot of your address space. Letting the OS cache is "free", both in terms of address space and memory. The OS will use pages for cache that it doesn't use otherwise anyway.
And lastly, doing your own private caching means that if the user launches three processes of yours, then the computer uses three times as much memory as it would need and takes (nearly) three times as long to load the data.
Using memory mapping would be the other extreme, in which case the OS would simply map the same memory in three different processes, thus it would not even do the memcpy from the cache.

Quote:

Original post by JonW Another useful technique for fast writing if anyone is curious, is to preallocate the file.

Note that you must run your program as administrator (or a user who belongs to a group that has the SE_MANAGE_VOLUME_NAME privilegue otherwise), or this will fail. Having to tell the user that you need admin rights is one huge downside.
I always find it unjustifiable when some random program wants admin rights, and with very few exceptions, it simply means that I'll not use the program.

You both have some good points. I tend to overthink performance concerns like this way too much...

As for the permissions issues, I *believe* you only need the administrator privileges to turn off file zeroing (as this is a requirement of C2 security). I haven't tried it, but you should be able to still preallocate with basic user privileges by using SetFilePointer and SetEndOfFile as mentioned. For SetLimit in .NET, I have no idea.