Dynamic maths library implementation

Lets say one wanted to implement a maths library which selected optimized math routines (class methods) for the system it ran on at run-time. It seems to me that the most logical way to implement such a system would be to use an abstract class (of a vecor or matrix) and then derive child classes for each system optimization and also a standard one w/ no hardware enhancements. Then, use a class facotry to spawn the proper object instance from the appropriate class. However, from what i read, this approach might not be the best due to the overhead that comes w/ virtual functions. I have read the other topics which suggest compiling different versions, but my main objective is to create a system that adapts at runtime. Anyone have any thoughts or suggestions as to the most clever and efficient implementation??