I'm working on a project to build 3D models from image sequences and video. I've already made considerable progress in that I have a program that can turn correspondence points into 3D models. Now I'm left with the problem of automatically generating those correspondence points (as opposed to picking them by hand). I've tried a basic gradient descent on the color information and that sort of worked, but it's unreliable and generally crude. If anyone on these forums has ever looked at optical flow or tried to solve the correspondence problem I would really appreciate being pointed in the right direction. I'm processing offline and I've got access to a condor set-up with 70 or so cores, so the algorithm doesn't need to be particularly fast. I'm also considering purchasing a book on the subject, but I wouldn't know what to get and with steep price tags I can really only afford one or two books at the moment.

Ive been out of the computer vision "game" for a while, but the Lucas-Kanade method used to be the most popular one by far.

Of course with all methods, beware of lighting, specular spots, and compression.

Using optical flow to generate apparent motion to estimate depth brings an age-old problem, tho. For the depth estimation to be accurate, you'll need big differences in the apparent motion. But if the apparent motion is large, optical flow estimation tend to be unreliable.

Thanks, I'll look into that method. I was hoping I could use optical flow for frames that are adjacent or close together and then follow a point through multiple frames (removing noise and whatnot) to get good points for 3D estimation, but then that's another age-old problem of turning an enormous amount of mediocre data into a useful amount of good data.

Do you need dense or sparse data?

I don't think I "need" dense since it works pretty well just triangulating a few dozen good points, but I was under the impression more points meant more detailed models. Is there something like feature tracking that would be better if I only needed sparse data? I'd like the program to be as general as possible, but I believe the project was pitched with the idea of modeling the inside and outside of buildings.

you could certainly do feature tracking if you only need sparse data. Dense data is possible, but the processing time would dramatically increase. I would recommend Shi and Tomasi feature detection, as they allow for good real-time requirements. OpenCV has a implementation that is easy to use.

For matching, I have used normalized cross-correlation in the past with gray scale images with great success. If the movement is guaranteed to be small, then you can limit the search space to a bounding box around the previous location. But, you should certainly start searching around the previous location to limit false positives.

Another, option for matching would be the Lucas-Kanade Feature Tracker. Again, I have used this will great success with Shi-Tomasi feature detection. Basically, it builds a pyramid of the image at different resolutions and starts the search at previous location of a feature.

Other popular feature detection algorithms are SIFT, FAST, SURF, etc... All have different benefits depending on requirements in speed, environments, lighting, etc.

Yeah, I was going to suggest SIFT feature tracking if you can live without dense data. The feedback I had is that it became quite popular and worked better than Shi&Tomasi. Man I can't believe "Good feature to track" is almost 15 years old already. I think I still have a paper copy somewhere...