Reflections of my thoughts... » Debugging

WinDBG: How to list call stack of all threads in managed dump/application?

@sarat — Wed, 16 Nov 2011 11:48:06 +0000

Even before analyzing the dump using !analyze with a crash/hang dump, I really wanted to see all threads call stack in a single shot. Mostly I use ~*kb to list the call stack with all threads for a native application/dump with WinDBG.

But WinDBG and .NET call stacks aren’t in a good sync. We will have to load SOS.dll or SOSEX.dll to get better help with debugging.

You can use !EEStack of SOS.dll to display call stack of all threads.

.load C:\Windows\Microsoft.NET\Framework\v2.0.50727\SOS.dll !EEStack

WinDBG – Visualizing STL containers and strings

@sarat — Tue, 16 Aug 2011 06:33:18 +0000

With the help of STL extension, this extension can’t visualize each and every containers or classes in STL, rather it displays the mostly used string, wstring, vector<[w]string>, list<[w]string>

for more information give !stl -? in WinDBG input or check in the help file.
See the example below and the output in the debugger console.

// SampleSTL.cpp : Defines the entry point for the console application.
// Compiled with Microsoft Visual C++ 9.0 compiler

#include "stdafx.h"
#include 
#include 
#include

#include 

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{

	string str1("Quick");
	string str2("Brown" );

	string str3 = str1 + " " + str2;

	vector vec;

	vec.push_back(str1);
	vec.push_back(str2);
	vec.push_back(str3);

	list lst;

	lst.push_back( str1 );
	lst.push_back( str2 );
	lst.push_back( str3 );

	for( vector::iterator iter = vec.begin(); iter != vec.end(); ++iter)
		cout << (*iter).c_str() << endl;

	cout <::iterator iter = lst.begin(); iter != lst.end(); ++iter)
		cout << (*iter).c_str() << endl;

	return 0;
}

WinDGB output

0:000> !stl str1
[da 0x16fb18]
0016fb18  "Quick"

0:000> !stl str2
[da 0x16faf0]
0016faf0  "Brown"

0:000> !stl str3
[da 0x16fac8]
0016fac8  "Quick Brown"

0:000> !stl vec
	Element 0
[da 0x558358]
00558358  "Quick"
	Element 1
[da 0x558378]
00558378  "Brown"
	Element 2
[da 0x558398]
00558398  "Quick Brown"

0:000> !stl lst
The list has 00000003 elements
	Element 0
[da 0x558400]
00558400  "Brown"
	Element 1
[da 0x558468]
00558468  "Quick Brown"
	Element 2
[da 0xffffffffcdcdcdcd]
cdcdcdcd  "????????????????????????????????"
cdcdcded  "????????????????????????????????"
cdcdce0d  "????????????????????????????????"
cdcdce2d  "????????????????????????????????"
cdcdce4d  "????????????????????????????????"
cdcdce6d  "????????????????????????????????"
cdcdce8d  "????????????????????????????????"
cdcdcead  "????????????????????????????????"
cdcdcecd  "????????????????????????????????"
cdcdceed  "????????????????????????????????"
cdcdcf0d  "????????????????????????????????"
cdcdcf2d  "????????????????????????????????"

The case of Visual Studio 2008 SP1 crash on Docking/Undocking and Switching from Debugging to Design View

@sarat — Fri, 12 Aug 2011 06:01:27 +0000

I’m using dual monitors with my laptop and I found that Visual Studio is crashing on docking/undocking window to/from other display. Although I am not do the same operation it was crashing again on switching back to design/code view from debugging.

I’m primary using IBM Clearcase as versioninig system and I have n number of issues ever since I started using with Windows 7. The same issue happened though I copied my source to local folder and debugging from there.

Simply took the dump and attached to WinDBG. !analyze -v had given me following call stack

EXCEPTION_RECORD:  002ad92c -- (.exr 0x2ad92c)
ExceptionAddress: 3fe083d0
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000008
   Parameter[1]: 3fe083d0
Attempt to execute non-executable address 3fe083d0

FAULTING_THREAD:  00002718

DEFAULT_BUCKET_ID:  SILENT

PROCESS_NAME:  devenv.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

NTGLOBALFLAG:  400

APPLICATION_VERIFIER_FLAGS:  0

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x2718 (0)
Current frame:
ChildEBP RetAddr  Caller,Callee
002af168 71bc5a99 (MethodDesc 0x714cc250 +0x79 System.Collections.Generic.Dictionary`2+ValueCollection+Enumerator[[System.Int32, mscorlib],[System.__Canon, mscorlib]].MoveNext())

CONTEXT:  002ad948 -- (.cxr 0x2ad948)
eax=59730146 ebx=00000000 ecx=01a72c50 edx=0000001d esi=01a72c50 edi=01a729c8
eip=3fe083d0 esp=002adc2c ebp=002adeb0 iopl=0         nv up ei ng nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010286
3fe083d0 ??              ???
Resetting default scope

WRITE_ADDRESS:  3fe083d0 

FOLLOWUP_IP:
msenv!CDockManager::PruneAllDeadPanes+45
5986cbf6 e97745f8ff      jmp     msenv!CDockManager::PruneAllDeadPanes+0x45 (597f1172)

FAILED_INSTRUCTION_ADDRESS:
+5789952f04fedcac
3fe083d0 ??              ???

PRIMARY_PROBLEM_CLASS:  SILENT

BUGCHECK_STR:  APPLICATION_FAULT_SILENT

LAST_CONTROL_TRANSFER:  from 5986cbf6 to 3fe083d0

IP_ON_HEAP:  3fe083d0
The fault address in not in any loaded module, please check your build's rebase
log at \bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

IP_IN_FREE_BLOCK: 3fe083d0

STACK_TEXT:
3fe083d0 unknown+0x0
5986cbf6 msenv!CDockManager::PruneAllDeadPanes+0x45
597eef9b msenv!CDockManager::SaveUI+0xb7
598122a5 msenv!CPane::TransitionToDesign+0x46
597f67a9 msenv!CVsShellDebugMgr::m_OnModeChange+0xe6
5976aca3 msenv!CVsUIShell::OnModeChange+0x21
5f4dbfa3 vsdebug!CDebugger::SetShellMode+0x9b
5f4dc04f vsdebug!CDebugger::OnModeChange+0x44
5f4dbeef vsdebug!CDebugger::SetDebugMode+0x69
5f4f2c93 vsdebug!CDebugger::OnExitBreakState+0xaa
5f53cab1 vsdebug!CDebugger::OnDebuggingStopped+0x6e
5f51bcb6 vsdebug!CDebugger::HandleEvent+0x8ae
5f4dbc19 vsdebug!CDebugger::Event+0xef
5f4dbaef vsdebug!sdm::CDebugManager::HandleEvent+0xb5
75eefc8f rpcrt4!Invoke+0x2a
75f54c53 rpcrt4!NdrStubCall2+0x2d6
75e9d936 ole32!CStdStubBuffer_Invoke+0xb6
75e9d9c6 ole32!SyncStubInvoke+0x3c
75e9df1f ole32!StubInvoke+0xb9
75db223c ole32!CCtxComChnl::ContextInvoke+0xfa
75db2131 ole32!MTAInvoke+0x1a
75db30fa ole32!STAInvoke+0x46
75e9de47 ole32!AppInvoke+0xab
75e9dcbb ole32!ComInvokeWithLockAndIPID+0x372
75e9e34c ole32!ComInvoke+0xc5
75db2ed2 ole32!ThreadDispatch+0x23
75db2e91 ole32!ThreadWndProc+0x161
762986ef user32!InternalCallWinProc+0x23
76298876 user32!UserCallWinProcCheckWow+0x14b
762989b5 user32!DispatchMessageWorker+0x35e
76298e9c user32!DispatchMessageW+0xf
75d7d6be ole32!CCliModalLoop::PeekRPCAndDDEMessage+0x4c
75d7d66d ole32!CCliModalLoop::FindMessage+0x30
75d7d57e ole32!CCliModalLoop::HandleWakeForMsg+0x41
75d7d633 ole32!CCliModalLoop::BlockFn+0xc3
75d71117 ole32!CoWaitForMultipleHandles+0xcd
5f513ed6 vsdebug!sdm::CDebugManager::WaitForEvent+0x8a
5f526aa0 vsdebug!CDebugger::WaitForDebuggingToStop+0x51
5f526fed vsdebug!CDebugger::TerminateSession+0x106
5f52711a vsdebug!CDebugger::StopDebugging+0x10a
5f4e8014 vsdebug!CVSDebugPackage::Exec+0x5f9
5974249f msenv!CVSCommandTarget::ExecCmd+0x937
59884617 msenv!FTranslateAcceleratorEx+0x4a3
597581c3 msenv!FTranslateAccelerator+0x49
59758045 msenv!MainFTranslateMessage+0x1a7
59757f2b msenv!CMsoComponent::FPreTranslateMessage+0x72
59757ec0 msenv!SCM_MsoStdCompMgr::FPreTranslateMessage+0xdd
5974152c msenv!EnvironmentMsgLoop+0x159
597fb9bd msenv!CMsoCMHandler::FPushMessageLoop+0x86
597fb94d msenv!SCM::FPushMessageLoop+0xb7
597fb8e9 msenv!SCM_MsoCompMgr::FPushMessageLoop+0x28
597fb8b8 msenv!CMsoComponent::PushMsgLoop+0x28
597fbe4e msenv!VStudioMainLogged+0x482
597f7561 msenv!VStudioMain+0xc1
2f6aaabc devenv!util_CallVsMain+0xff
2f6a78f2 devenv!CDevEnvAppId::Run+0x11fd
2f6a7bf4 devenv!WinMain+0x74
2f6a7c68 devenv!License::GetPID+0x258
765a1114 kernel32!BaseThreadInitThunk+0xe
7791b429 ntdll!__RtlUserThreadStart+0x70
7791b3fc ntdll!_RtlUserThreadStart+0x1b

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  msenv!CDockManager::PruneAllDeadPanes+45

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: msenv

IMAGE_NAME:  msenv.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  488f2b9f

STACK_COMMAND:  .cxr 002AD948 ; kb ; dds 2adc2c ; kb

FAILURE_BUCKET_ID:  SILENT_80000003_msenv.dll!CDockManager::PruneAllDeadPanes

BUCKET_ID:  APPLICATION_FAULT_SILENT_BAD_IP_msenv!CDockManager::PruneAllDeadPanes+45

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/devenv_exe/9_0_30729_1/488f2b50/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1

Followup: MachineOwner
---------

Luckily a Google search with the call stack has returneda single relevant result about this issue in Microsoft Connect.

Visual Studio 2008 IDE Crashes Returning to Design Mode When Stopping Debugging Native C++ Code

Microsoft has released a patch to resolve this issue. Download

Tools: How to monitor log files in real-time?

@sarat — Fri, 29 Jul 2011 13:49:35 +0000

It’s always painful for users that to see the log file in real-time. Probably we will do some logging mechanism and then the file will be updated by application and we’ve no way to track what’s happening.

Windows programmers rely on traces (OutputDebugString, Debug, Trace classes in .net) etc. which can be monitored in debugger output window or using handy tool called DbgView. But the pain here is we’ve to add the code again and take the build to see things in action.

What if we can do the same stuff with a tool? BareTail is real-time log file monitoring tool with lot of filtering options for highlighting and storage. It also supports multiple tabs, UNICODE/Non-Unicode files etc. Also it’s a standalone application where no installation required. So it’s suitable for using in the target machines where the installations of the softwares are restricted.

Using Debug Trace with C# (.NET)

@sarat — Thu, 28 Jul 2011 13:55:24 +0000

In C++, we’ve TRACE and OutputDebugString diagnostic services get the debugger output or program traces. How we do this in .NET?

System.Diagnostics.Debug.WriteLine("Using Debug.WriteLine");
System.Diagnostics.Trace.WriteLine("Using Trace.WriteLine");

Debug.WriteLine will be available only in the debug builds/while debugging. While Trace.WriteLine can work in Debug/Release mode. You can also see the output using handy SysInternals tool called DbgView

LiveKD error: Could not resolve symbols for ntoskrnl.exe: MmPfnDatabase

@sarat — Wed, 16 Feb 2011 18:49:06 +0000

When you start LiveKD with kernel mode debugging, you may see an error as below.

Could not resolve symbols for ntoskrnl.exe: MmPfnDatabase

The reason is LiveKD unable to resolve the proper symbol for the ntoskrnl.exe. I simply loaded an old system dump file (check %systemroot%\Minidump) with WinDBG and let it download the symbol files itself. If the symbol files are already downloaded and corrupted. Just retry using .reload command. It should work fine. (I’m not sure how to make WinDBG to directly cause downloading the file). Alternatively if you can one of these methods to generate a kernel mode dump

Magic Debug Values

@sarat — Fri, 31 Dec 2010 06:33:31 +0000

Magic debug values are specific values written to memory during allocation or deallocation, so that it will later be possible to tell whether or not they have become corrupted, and to make it obvious when values taken from uninitialized memory are being used.

See a list of popular magic debug values (retrieved from Wikipedia)

Code	Description
..FACADE	Used by a number of RTOSes
8BADF00D	Used by Apple as the exception code in iPhone crash reports when an application has taken too long to launch or terminate.
A5A5A5A5	Used in embedded development because the alternating bit pattern (10100101) creates an easily recognized pattern on oscilloscopes and logic analyzers.
ABABABAB	Used by Microsoft‘s HeapAlloc() to mark “no man’s land” guard bytes after allocated heap memory
ABADBABE	Used by Apple as the “Boot Zero Block” magic number
ABADCAFE	A startup to this value to initialize all free memory to catch errant pointers^{[clarification needed]}
BAADF00D	Used by Microsoft‘s LocalAlloc(LMEM_FIXED) to mark uninitialised allocated heap memory
BADBADBADBAD	Burroughs large systems “uninitialized” memory (48-bit words)
BADC0FFEE0DDF00D	Used on IBM RS/6000 64-bit systems to indicate uninitialized CPU registers
BADCAB1E	Error Code returned to the Microsoft eVC debugger when connection is severed to the debugger
BADDCAFE	On Sun Microsystems‘ Solaris, marks uninitialised kernel memory (KMEM_UNINITIALIZED_PATTERN)
BEEFCACE	Used by Microsoft .NET as a magic number in resource files
C0DEDBAD	A memory leak tracking tool which it will change the MMU tables so that all references to address zero
CAFEBABE	Used by both Universal Mach-O binaries and Java .class files
CAFEFEED	Used by Sun Microsystems‘ Solaris debugging kernel to mark kmemfree() memory
CCCCCCCC	Used by Microsoft‘s C++ debugging runtime library to mark uninitialised stack memory
CDCDCDCD	Used by Microsoft‘s C++ debugging runtime library to mark uninitialised heap memory
CEFAEDFE	Seen in Intel Mach-O binaries on Apple Inc.‘s Mac OS X platform (see FEEDFACE)
DDDDDDDD	Used by MicroQuill’s SmartHeap and Microsoft’s C++ debugging heap to mark freed heap memory
DEADBABE	Used at the start of Silicon Graphics‘ IRIX arena files
DEADBEEF	Famously used on IBM systems such as the RS/6000, also used in the original Mac OS operating systems,OPENSTEP Enterprise, and the Commodore Amiga. On Sun Microsystems‘ Solaris, marks freed kernel memory (KMEM_FREE_PATTERN)
DEADDEAD	A Microsoft Windows STOP Error code used when the user manually initiates the crash.
DEADF00D	Used by Mungwall on the Commodore Amiga to mark allocated but uninitialised memory ^[12]
DEADFA11	Used by Apple as the exception code in iPhone crash reports when the user has force-quit the application.
EBEBEBEB	From MicroQuill’s SmartHeap
FADEDEAD	Comes at the end to identify every AppleScript script
FDFDFDFD	Used by Microsoft‘s C++ debugging heap to mark “no man’s land” guard bytes before and after allocated heap memory
FEE1DEAD	Used by Linux reboot() syscall
FEEDFACE	Seen in PowerPC Mach-O binaries on Apple Inc.‘s Mac OS X platform. On Sun Microsystems‘ Solaris, marks the red zone (KMEM_REDZONE_PATTERN)
FEEEFEEE	Used by Microsoft‘s HeapFree() to mark freed heap memory

See Wikipedia for more details – http://en.wikipedia.org/wiki/0xDEADBEEF#Magic_debug_values

Unresolved breakpoints and debugger performance

@sarat — Mon, 06 Dec 2010 06:08:38 +0000

With WinDBG, You might have experienced slow symbol loading, and poor performance on attaching process to debugger, underperforming commands like, !analyze, ~*# (list all threads’ call stack) etc.

One of the core reason for poor debugger performance is unresolved breakpoints in the workspace. If a breakpoint is set for a routine name that has not been loaded, the breakpoint is called a deferred, virtual, or unresolved breakpoint. (These terms are used interchangeably.) Unresolved breakpoints are not associated with any specific load of a module. Every time that a new application is loaded, it is checked for this routine name. If this routine appears, the debugger computes the actual coded address of the virtual breakpoint and enables the breakpoint. But this operation is too expensive. Try to avoid unrsolved break points and always use the right module names before putting the break point. (e.g bp ntdll!RtlRaiseException )

Please remove the unnecessary breakpoints with the help but breakpoint commands. Some of the most used commands are listed below. For more help, please see WinDBG help file

bp – set breakpoint
bl – list the break points
bc – clear all or particular breakpoints
.bpcmds – list the break points with the commands which used to create them.