Disable Rivatuner/Afterburner or other OSD for directX or OpenGL C++ program

I think it depends on how the application is designed. Unreal is different from Unity and Godot that they use a GUI library like for example WPF while Unity and Godot (I don't know but I guess it is the same for Godot) are rendered on top of the engine, so technically you have a game (window) with some additional editor features enabled.

I guess Afterburner is hooking into the process by some kind of GraphicsCard / Hardware related trigger. Normal applications like VisualStudio or a Browser use the OS internal rendering interfaces provided by GDI+ (on Windows) for example and so might be different in how they are handled by the graphics card.

But at the end you won't know except you can get the source code of Afterburner. I don't think it is something you can just turn on/off by some API call.

However, you can do a (more or less) simple test to see if it still happens if you embedd the render context into some GDI code. You can for example use C# and p/invoke to pass the window handle of a control to setup code of the render context for OpenGL and see if it still happens that Afterburner attaches the overlay to it

also view port in both normal and play mode dosent have osd.

Is it means it is not Opengl/D3d

In cryengine 5.5 and 5.4 that I tested, Just viewport has OSD

What about an OpenCL custom renderer?

Or Qt and other stuff like that

The better questions is: Why do you care?

Is it worth the work to just “turn off” Afterburner to find a way to prevent it from attaching it's overlay to your Editor instead of investing time in the Editor/ Engine itself?

If it just effects your visual view, turn it off