StarMire: I see you're calculating the distance from the way-point to the (infinite) line defined by the enemy's original position and its current position. But what's the actual test to decide whether you've reached the way-point or not?
This is just looking at the relevant line segment, assuming you have passed the waypoint.
[attachment=24224:DNewOldWay1.gif]
If you have not passed the waypoint, the angle assumptions are wrong, you're right.
So, one should probably check to see if one has passed the waypoint first, either as per your method or the one I mentioned before, adjusted for near vertical or horizontal movement (which I think can be done algebraically rather than as a condition test), and if passed, this method will tell you if it ever came close enough in said passing.
Assuming it didn't turn a really quick circle around the point and still avoid it.
Your method of polar acceleration should help dampen orbits though.