Setting up Visual Studio for Shader Debugging

This tutorial assumes that you already have the following items set upon a machine with Windows XP installed;

    1) Visual Studio.NET is installed on your PC

    2) The DX9 SDK is installed on your PC.

    3) The debug runtime is selected

Only Windows XP is supported, but there are plans to support Win 2000 in the future.

Once the SDK is installed, you'll need to select the debug runtime and enable shader debugging. From the Control Panel, select DirectX, then select the directed tab. Make the required changes. These are shown in the image.

Once that's done, you'll need to have a source file that contains the shader, (if you don't have the source, try this) and either use the shader compiler (vsa for vertex, psa for pixel, and fxc for HLSL) and specify the /Zi flag, or use the D3DX helper routines (D3DXAssembleShaderFromFile or D3DXCompileShaderFromFile) with the D3DXSHADER_DEBUG flag specified as an argument. You should omit any optimization flags when compiling for debugging. Then example below

DWORD dwFlags = 0;
#if defined( _DEBUG ) || defined( DEBUG )
dwFlags |= D3DXSHADER_DEBUG;
#endif
// Assemble the vertex shader from the file
if( FAILED( hr = D3DXAssembleShaderFromFile( strVertexShaderPath,
NULL, NULL, dwFlags, &pCode, NULL ) ) )
{
return hr;
}

Once that's done you can start debugging.

You can launch the process in the debugger directly, or attach to a D3D process.

To launch the debugger with the process, Select Debug from the menu, then Direct3D, then Start D3D. (Note that Ctrl+Alt+F5 is the default keypress for this).

To launch the debugger on a process that's running (as one started from the menu Debug, then Start Without Debugging), you select Debug, then Processes (to open the Processes menu), then select the Direct3D app you want to debug and press the Attach button to open the Attach to Process dialog, shown below.

Make sure the Direct3D button is selected. You also have the Native button selected if you want to debug your C++ or C# code as well. You can also attach to a process running on another computer as well using this method by entering the name of the computer the process is running on.

You cannot debug a HAL (hardware) device if you've selected a pure device interface. In the debug output when your device is loaded you'll see the device that's selected. If it says "Hal HWVP Pure device selected", you can't debug anything. If it's a non-pure device then you can debug only vertex shaders, not pixel shaders.

To debug both pixel and vertex shaders, you need to select the reference device (REF) (either pure or not, it makes no difference). In most of the SDK samples you can change the device on-the-fly by selecting F2 and changing the Device Type to D3DDEVTYPE_REF.

You can then bring up the shader file and set breakpoints. Once a breakpoint has been reached, you can examine/change registers,

If you don't have the shader source code.

If you don't have access to the source for the shader, you can start debugging the process and create a new breakpoint. On the New Breakpoint dialog box, set the Language to Direct3D Shader and the Function to "BeginScene" as shown below, then press OK.

When you hit the breakpoint you'll get a dialog complaining that is doesn't have the source code for the current location. Close the dialog and either press F10 or select Debug, then Step Over. You'll be in the first shader to get run.