Debugging shader in .FX files

You might want to try nVidia's FX Composer. It lets you edit and debug your FX files outside of your code. That way you'll know you have a working shader before wrapping any C++ code around it. You'll also know that any bugs introduced at that point are likely not to be in your shader.

Yes, you can debug vertex and pixel shaders with HLSL. You have to do the following:

(1) If you just want to debug vertex shaders, specify software vertex processing when creating the device (use D3DCREATE_SOFTWARE_VERTEXPROCESSING). You should also specify D3DXSHADER_FORCE_VS_SOFTWARE_NOOPT when you create your effect with D3DXCreateEffectFromFile().

(2) If you want to debug pixel shaders, you must create a reference device (use D3DDEVTYPE_REF). You should also specify D3DXSHADER_FORCE_PS_SOFTWARE_NOOPT when you create your effect with D3DXCreateEffectFromFile().

(3) If you want to debug vertex and pixel shaders, create a reference device. You should also bitwise or (|) D3DXSHADER_FORCE_VS_SOFTWARE_NOOPT and D3DXSHADER_FORCE_PS_SOFTWARE_NOOPT together when creating the effect.

Specifying D3DXSHADER_FORCE_*_SOFTWARE_NOOPT improves debugging, since it enables source level debugging, prevents instruction reordering, and prevents dead code elimination. It also forces the compiler to compile against the next highest shading model, to ensure that the unoptimized shaders do not exceed the shader model limiations.

I wouldn't recommend using an IDE like rendermonkey or FX Composer to fully develop shaders. It's fine for working out the pipeline of the shader and whatnot, but due to the implementations in the IDE's they don't accurately show you how it's going to be implemented in the game engine. I usually just develop a framework in the IDE and then use a quick program of my own where I can edit the code and see exactly how it's working in the proper DirectX coded environment.