One way hashes

You need to ask yourself how important this really is. My hunch is that you don''t need anything close to MD5 or SHA, and that you could do an excellent job by using something like Zobrist hashing.

Zobrist hashing will be significantly faster than MD5 or SHA, and it gives good "spread out" keys, meaning that if you change one bit in the entire gigantic file, your key will be drastically different.

A simple way to use zobrist keys for your purpose would be to set up a 256-element array, and in each of those elements, store a 32-bit random number (or you can use 64-bit if you''d like). Then to compute the key, take each byte in the file and use that byte as an index into the random number array, and XOR the value onto the key. Each byte will XOR a different random number, and it will change the key significantly.

You might want to try using a 16-bit table (65,536 elements) instead of the 8-bit table (256-elements), since that would give you better hash keys. I would go with a 16-bit table, using 64-bit keys, and you should be set.