Detecting software fallback?

You cannot. The OpenGL specs do not provide a method to detect this.

Your best bet is to check your shader logs (GetShaderInfoLog(), GetProgramInfoLog()) for keywords like "fall back" or "software" when compiling / linking your shaders.

Well, there are roughly two types of "software fallback" :
1- your system is running a software GL implementation (most likely MESA) which means that all your OpenGL calls will be performed in software,
2- you are running a GL driver which performs most operations on the graphics card, but few operations are not supported in hardware and the driver uses a software path for them.

The former is pretty easy to detect : once your GL window is initialized, call glGetString(GL_RENDERER) which returns a char*, and check the contents : if it contains the word "Mesa", you're rendering everything in software. There are several reasons if this happens, including but not limited to bad graphics driver. Most of the time installing the latest drivers for your graphics card will solve the problem.

The latter is much more complex to detect and even more complex to solve (if ever possible). First of all, as Fiddler pointed out, in OpenGL there is no way to query directly the driver to know which features are available in hardware and which ones in software only. To make sure a particular point is running is software, you have to detect yourself the bottleneck and run performance counters and things alike. To make things even worse, some features may be available in hardware up to certain limits. For instance some very old GeForce used texture units to implement clip planes, in that case if your number of texture units used + number of clip planes were lower or equal to the total number of texture units available into the graphics card, everything was running smoothly, otherwise you would fall back to software for a reason that was almost impossible to comprehend. In other words, this field may be extremely complicated and far beyond the amount of time that programmers can afford to spend. Use common GL commands as much as possible. Avoid functions that are known to behave poorly, such as glReadPixels, and avoid features that are known not to be used anymore, such as color tables. Test your engine as often as you can, and if the framerate seems to drop dramatically when you introduce a new feature, look into it first but keep in mind it may not be the only reason behind the framerate drop, it may just be an element amongst a heap of combinations.

The simplest solution is measure the FPS. If it is below 10, I would say it is way to slow or pick whatever number you like.

Thanks for the replies. It's sad to see that there is no easy way. I'm testing my engine on many computers, operating systems and GPUs, and even on some new CPU, OS and GPU combinations it's clearly falling into software mode. I must test it more with/without shaders/VBOs/stencil shadows.

Re vincoof: Mesa does not automatically imply software rendering, for example Intel's 3D hardware acceleration in Linux is implemented by Mesa, and I think that some Radeons are also hardware accelerated through it.

Unfortunately, there hasn't been an option to disable software fallback (by software fallback, I mean running the VS or FS and rasterizer in software) in GL. I am not sure but I think with GL 3.2, software fallback are not allowed.

Furthermore, software fallbacks are due to certain things such as exceeding the GPUs shader limit or perhaps certain modes such as clipping planes or by having POLYGON_SMOOTH, LINE_SMOOTH and/or POINT_SMOOTH enabled (effect ATI).
And I'm talking about Windows.
For Linux, I think the pixelformat info has some sort of flag that tells you which one risks giving you slow speed. Not sure, I'm more of a Windows user.