Lightweight file locking?

Opening a file only prevents the file from being written to, why don't you just change the attribute on the files to be read-only?

Quote:

Original post by AnAss but I need to do this for potentially many thousands of files at a time.

What exactly are you trying to accomplish?

I have a data set stored in many files. I want to prevent any program from modifying any one of the files while I am processing the data set (one file at a time). My test data set has 20,000 files in it.

If I set the attribute to read only, and for some reason my program were to fail to reset the attribute when it is done, the files would be stuck with the read only attribute. If I use open file handles, the OS cleans up the handles should anything happen to my program. Also I don't think the read only attribute prevents the files from being deleted even in windows explorer, and many programs offer the ability to overwrite read only files anyways.

Quote:

Original post by AnAss I have a data set stored in many files. I want to prevent any program from modifying any one of the files while I am processing the data set (one file at a time). My test data set has 20,000 files in it.

Pre-Vista does not provide required transactional semantics which would allow this.

Locking/marking files without transactional support does not solve anything at all. If files are independent, then there is no need to lock them all. If they are dependent, then you need an atomic lock over all of them.

Consider what happens if you are opening/locking/marking 20,000 files, and external application changes one of them (after marking starts, before the file is marked).

To get around this, you would need to first register file notification (whatever the API is). Then, start locking/marking files. If you receive a change notification, restart the locking process (if that makes sense, if that is really the expected outcome). This process may never lock all files.


But until you determine what exactly defines atomicity, you might as well process the files individually - it will have same level of reliability.


But non-atomic locking/marking will not provide you with reliable result, so first determine what exactly makes for a single computational unit, and how changes to files affect that.


Alternate solution would be to use a solution suited to problems like this, either versioned file systems, transactional file system, version or content management system, or redesigning the data producers to avoid the issue altogether.

That makes sense, but even though it isn't totally atomic, its still better to have a window of 30 seconds for another program to modify a file, instead of many hours/days.

As for the OS/filesystem/dataset format, those are out of my control.

Quote:

Original post by AnAss That makes sense, but even though it isn't totally atomic, its still better to have a window of 30 seconds for another program to modify a file, instead of many hours/days.

That depends on how important the results are.

People who did the global warming studies wouldn't bother locking at all, for example. People on Wall Street however can perform 100,000 automated transactions in 30 seconds and earn or lose millions.


Regardless of time, it still doesn't answer the question. Let's say you have files A, B and C. Somewhere along the line, B gets modified. How do you know that original B was the right one? How do you know that new B is invalid? How does it affect the results?


This is probably the worst kind of reliability issue, since your application will make guarantees it cannot uphold.

How big are these files anyway?

The files in a particular data set are all exactly the same size, 10 to 50 MB for each file.

I realize not having an atomic lock is not ideal, but really, as long as I lock all the files in my data set before I begin processing, it is OK. If someone were to be modifying the files immediately beforehand, either they would have just finished modifying the last file they will modify, in which case, the data set is up to date and OK, or they will try to modify another one of the files soon, and they will get an error about the file being locked and know something is up.

So, I really just need a lightweight way to lock many files at a time.

For reference, the method I currently use is to only lock the most recently accessed 100 files in the data set! (by keeping the files open) That is obviously terrible and something I'm not proud of, but until I find a lightweight way to lock many files, its the best I've got. This technique prevents the most egregious mistake, which would be to use the same data set for input and output (overwriting the input data set with the processed data set).

Opening a handle to a file is the only way the kernel knows you've got an "interest" in it. So anything else you do cannot be automatically cleaned up if your process crashes.

Having said that, I think simply setting the read-only flag is probably "good enough". Most programs will honour the read-only flag, Explorer will give you a warning that the file is read-only, etc - that should be a warning to users that something is amiss.

In the event that your program crashes, you can implement an explicit "clean-up" command that resets everything (that's what Subversion [which has to process lots of files in your working directory] does, for example)